TRQ Research

The online version of Timed Repetitive Quizzes (TRQs) was introduced to an Organic Chemistry class at Lee University in Fall 2005. After several years of amazing results in chemistry, TRQs were used in a variety of subjects and educational levels. The research data collected during these implementations show substantial improvement in test results and course grade as described below. My deepest appreciation goes to Dr. Paul Conn, President Lee University (1986-2020), for his cooperation, encouragement and financial support that funded four TRQ research projects:

Multi-discipline (2010 and 2011)
Mongolian language (2014)
Spanish language (2016)

Organic Chemistry (2006-2017)

The American Chemical Society (ACS) Organic Exam has been given as the Spring Final for Organic Chemistry II at Lee University since 1991. The class average on the ACS Organic Exam (1991-2005, 248 students) was 41.0 percentile.

After TRQs were introduced (20% of the course grade), the class average (2006-2017, 389 students) was 59.4 percentile. Click the Play button in the interactive graph below to start the animation . . . . stop at any point by clicking the Stop button.

While a 19.4 percentile increase is phenomenal, the effect is actually greater when the Spring semester withdrawal rates are considered . . .

37 students withdrew out of 248, 14.9% from 1991-2005
3 students withdrew out of 389, 0.77% from 2006-2017

An unexpected outcome was the fact that TRQs provide a significant benefit to the "prepared" students . . .

13 students, 5.2%, scored above the 90^th percentile from 1991-2005
46 students, 11.8% scored above the 90^th percentile from 2006-2017

This 27-year longitudinal study provides strong evidence that TRQs are significantly beneficial to all students.

Lee University (2010-2011)
In Spring 2010, Seven classes participated in a research experiment where five sets of TRQs were assigned for each chapter. The TRQ assignments were 10% of the course grade and no review TRQs were assigned. In each subject, the Instructor taught two sections of the class . . . the TRQs were used in one of the sections.

Marketing
Physical Science
Environmental Science
Sociology
Elementary Algebra
Rise of Europe
Foundations of Western Culture

There were 282 TRQ students and 420 non-TRQ students (702 total students). The TRQ students had an average increase of 3.2 points on each exam and the final exam. Additionally, there were 8.7% more A's in the TRQ classes.

In Spring 2011, the research experiment was duplicated with the following classes . . .

Marketing
Physical Science
Environmental Science
Sociology
College Algebra
Contemporary Mathematics

There were 235 TRQ students and 236 non-TRQ students (471 total students). The TRQ students had an average increase of 3.6 points on each exam and the final exam. Additionally, there were 8.4% more A's in the TRQ classes.

An unexpected discovery was that 41% of students (212/517 students) did not earn more than 50% of the TRQ points . . . if you keep taking the TRQs, you can earn 100% of the TRQ points. While a 3.2 and 3.6 point increase is good, we wanted to know the full effect of TRQs with full participation. So, we pivoted and implemented TRQs in a 6^th grade math class where higher levels of student participation were possible because students were monitored as they worked on the TRQs during the class instructional time.

6^th Grade Math (2011-2013)
In 2011 - 2012, TRQs were used by 78 students in a 6^th grade math class (Ocoee Middle School (OMS), Cleveland TN) for 20% of the instructional time - the teacher condensed her previous classroom activities into the remaining 80%. The students showed a 12.8

percentile increase in the state's end-of-year exam (TCAP). The fourth row in the image (right) shows that 43.7% of the students arriving from the 5^th grade were classified as Proficient / Advanced , the remaining 56.3% were Basic or Below Basic . After using the Math TRQs for their 6^th grade year, 63.4% of these test students performed at the Proficient / Advanced level on this exam - an increase of 19.7% .

In 2011 - 2013, the TRQs were used in "Response To Intervention (RTI)" classes (6^th, 7^th and 8^th grades at OMS) comprised of only Basic Math Students as determined by their previous year's TCAP score in Math. Students attended the RTI class for 45 minutes every day for one semester where they exclusively used TRQs. Of the 557 students enrolled in the RTI classes (2011 - 2013), 197 students (35.4%) moved up to the Proficient / Advanced level on the TCAP exam at the end of the school year.

Mongolian (2013-2014)
In December 2013, a preliminary study was conducted with two students where they were tasked with learning two sets of 50 Mongolian words (100 total Mongolian words). Mongolian words were chosen for two reasons:

over 500 Mongolian TRQ questions had been created for missionaries to Mongolia
participants in the study had no prior knowledge of any Mongolian words

Students spent 2 hours on the first 50 words using "their best method" of study, took a quiz, took a 1-hour break, and then spent 2 hours on a different set of 50 words using the TRQ method. The next day the students studied the words again (1 hour each set) with the method used the previous day. The assessment's criteria were . . .

the Mongolian words had 4-7 characters
each set of 50 words had a total of 250 characters
the format was fill-in-the-blank (mastery credit was awarded for correct spelling)
the English text was given and the students entered the phonetic spelling of the Mongolian word

The results show an intial advantage when studying with TRQs on a "single exposure" . . . 99% correct (TRQ) vs. 89% correct. Also, the 1-day forgetting curve is much steeper (89%→53%) for the students' "best method" compared to 99%→89% for the TRQ method. The 1-hour review at the 1-day mark significantly improved the "best method" results (89%→99%) making the two methods "appear" equal. However, in the 4 weeks that followed, it was the TRQ method that had a much flatter, less forgetting curve.

These results led to a research study (April 2014) where 19 students used their "best method" to learn 50 Mongolian words and then the TRQ method 24 hours later to learn the second set of words. The students were paid volunteers from an Organic Chemistry II class (mostly Pre-Med students) and a Learning and Cognition class (an upper-level Psychology class). The students took a fill-in-the-blank assessment at the end of 2 hours, 1 day and 1 week. Their answers were marked either correct or incorrect - spelling counted.

For the TRQ method, the students were able to correctly answer 14% (11/78×100) more words on the initial assessment, 16% (11/69×100) more words after 24 hours and 44% (16/36×100) more words after 1 week than the non-TRQ method at the corresponding times.

It is worth noting that a 2-hour, single-exposure use of TRQs is not the prescribed route of implementation, but we were interested in a direct comparison of the students' best method of studying to the TRQ method. The 53.5% loss, after 1 week, of words learned using the student's best method and 41.5% of words using the TRQ Method, leaves little doubt that repetitive exposure to the information is essential for moving more information into long-term memory.

Why haven't I heard of TRQs?
You are probably thinking "these results are phenomenal . . . why haven't I heard of TRQs?"

The "secret sauce" to body building, weight loss, learning a new language, etc. is not a secret - it's hard, uncomfortable work . . . . but, very rewarding over time. The key to academic success is the same. We are building the Let's Go curriculum for Grades 7 - 12 because these students are mature enough to work online and because they are required to be in class. How much easier would success come in the gym if students were required to be there for 6 hours every day? The key to academic success is to apply an approach that forces students to develop academic muscles. The applied force is simply an expectation that they quickly complete work that they can do . . . and you only give them work they can do. This was most notably implemented in the mid 1950's by the Harvard professor, B. F. Skinner, using The Teaching Machine. There are many similarities between his Teaching Machine and TRQs . . . you will find his video enlightening (click image).

Back to the question "why haven't I heard of TRQs?" The short version of a long story (18 years) has two chapters . . . the teacher and the student.

teacher - most educators can't comprehend the value TRQs bring to the novice learner. I was like that in chemistry - I was a student of chemistry in high school (2 years), college (4 years), graduate school (3 years) and by 1997 I had taught college chemistry for fourteen years (7 in graduate school and 7 at Lee University). I could not comprehend why some of my students could not "readily learn" the facts that I just somehow "knew" . . . . if they did, we could have a lot more fun in chemistry. I began to believe that I just had a "mind for chemistry" . . . "a gift" . . . and only "like-minded" students could succeed in chemistry.

Then I enrolled in Spanish class. I actually wanted to succeed, but of course I didn't have a "mind for Spanish" . . . I had a science mind. Instead of focusing on my deficiencies, I began doing the work I could do. It turns out that work, despite your "mind type", will build academic muscles. My re-entry into an academic area as a novice forced me to see my chemistry courses as my students were seeing them. As a chemistry expert, I found great personal enjoyment in group work, discussion, explanation, collaboration, critical thinking, writing and presenting . . . . I envisioned that it would be mind-numbing anguish to repetitively recall basic facts . . . . and it would be for an expert. What I did not realize was that I had repeatedly (over the course of two decades) recalled basic facts. If I wanted my students to remember chemistry after my course and to get to more of the "fun part" during my course, then the repetition work must be concentrated . . . TRQs made that possible. TRQs helped my students develop their academic muscles much like a weightlifter builds real muscles during concentrated effort. My Spanish class taught me that novices simply can't do expert work (the work teachers find fun and exciting) until time and energy are invested in the fundamentals of learning.

Unfortunately, it seems that the lesson I learned is best learned through the experience of being a novice learner again. Several math and science teachers have used TRQs to learn Spanish and this experience has led them to use TRQs in their teaching discipline. Other teachers have used TRQs in their classes because of a group adoption . . . over time, these teachers see the effect TRQs have on student learning. Teachers outside these two groups are either unable to grasp the necessity for students to acquire basic knowledge or they are unwilling to intrude into that area of learning . . . . "students should want to learn." My conversations with this last group of teachers include the following comments . . .
- A foreign language teacher said to me "I would rather a student come to my class with ZERO knowlege than to have memorized a thousand words." She made quite a distinction between memorization and "knowing" . . . memorization is bad, knowing is good. She also said she "would never tell me how to teach chemistry."😄 TRQs don't teach, teachers teach. My goal in expanding TRQs to other subject areas is to prepare and equip students so that the teacher can take them further into the fun.
- One professor "didn't want students to regurgitate the textbook or my notes." It was said as if one could not critically think about a topic if they could also regurgitate a large volume of facts about the topic. The oddity of this statement is that experts can regurgitate fact after fact (ad nauseam) in their discipline with the only obvious critical thinking being the order in which they deliver the facts. Helping novices attain the skill of experts is the role of a teacher . . . . if experts "know" facts, then the novice should be acquiring facts.
- Another professor said that "TRQs are too much assigned work" and that "students should want to learn". In the "real world", students do assignments, not suggestions. If you want students to spend 6 hours / week learning for your class, suggestions like "read the chapter" and "know the information in Tables 1.3 and 1.4" will not create the expected results. Imagine a university coach that does not have required activities (practice, strength training, early morning runs, etc.) because "an athlete should want to do those things." I'm sure their season would not be successful . . . nor their future employment. Let's coach students to academic success.
- The Director of Secondary Education told me that "20% of instructional time is a lot to give up". This was in response to the 19.4% increase in the Proficient / Advanced levels for the 78 6^th grade math students. I had no response at the time, but upon reflection I surmise that there is a perception that nothing can be better than a teacher. I actually agree if the student is ready for higher order thinking / learning. But, for the student needing to build foundational knowledge in a subject, the teacher is no match for TRQs.
- Another teacher "liked" the TRQs, but some negative comments on the Student Course Evaluation prompted her to tell me "I will probably use them after I get tenure." This is a great place to talk about students and TRQs.
student - imagine weightlifting is a required class that all students must take. The first day is the hardest day - the most complaining, explaining and refraining. The second day is the next hardest day . . . . the 10^th day is harder than the 15^th day, etc. Now, if you ask these same students on the 60^th day how they like the class, most will say "it's a lot of hard work, but look at my muscles."

The most frequent comment on my student evaluations after implementing TRQs can be paraphrased as "as much as I hate to admit it, TRQs really work."

Some teachers that have "used" TRQs only required 2-3 sets because they didn't want to give "too many assignments". Employed in this fashion, TRQs are simply hard and ineffective . . . it's like two days of bench press (Chapter 1) followed by two days of arm curls (Chapter 2), followed by two days of squats (Chapter 3), etc . . . . without ever repeating these exercises. As a teacher (coach) you have to daily "stretch" your students with activities they can do so that over time they can achieve unimaginable results. Humans have a natural resistance to "testing" because it reveals deficiencies. Research has shown that instead of testing themselves while studying, students prefer to read and re-read. After a couple of passes through the notes, they become familiar with the material and develop a sense of accomplishment. This feeling vanishes quickly during the test when they have to RECALL information - either to enter it directly, or to arrange information to synthesize an answer. The unfortunate reality is that our distaste for testing locks us in a cycle that delays preparation, chooses inadequate preparation techniques and leads to failure. With failure comes explanations . . .
- I just can't get trigonometry.
- I'm not good in math.
- I'm better at essay tests.
- I have a language processing disorder.
- I have test anxiety.
. . . that create limitations.

Timed Repetititve Quizzes are "tiny tests" performed daily to mentally stretch students with a scientifically-proven study technique that produces long-term results.

"Why haven't I heard of TRQs?" Teachers are learners that have reached the expert level. The "new" learning in their discipline is "fun" - the activities are fun and learning the occasional "new fact" is fun because it fits in a perfect place in their mental schema.

Learning is Fun!

Students, however, struggle to place each "new fact" on their blank slate.

They must constantly reaffix facts. Then, they begin making connections and grouping related facts. To accomplish this work in the shortest amount of time requires "testing" - the attempted retrieval of a fact from long-term memory. This period of mental sweat is followed by "aha moments" where a connection is made or a "new fact" fits perfectly in the student's limited schema.

Learning has some Fun moments!

It seems that in many educational systems, the Learning is Fun approach does not acknowledge the hard work a novice must invest before learning becomes fun. Our experience with five different educational systems has convinced us that TRQs can only thrive and grow in a new educational system. So, we are building a TRQ-based curriculum and taking administrative control so we can educate our children with an approach that changes their long-term memory.