March 1998 — Volume 3, Number 2
Usefulness and Enjoyableness of Teaching Materials as Predictors of On-task Behavior
Department of English
City University of Hong Kong
This article reports on a small-scale study that investigated the relationship between on-task behavior, usefulness of materials as perceived by learners, and enjoyableness of materials. It was hypothesized before the study that learners spend a greater proportion of the lesson on task (that is, actively engaged in the task that they were set) when they see the lesson as useful and/or enjoyable.
Data were collected daily over one term in two South Korean university EFL classrooms. The correlation between on-task behavior and usefulness of materials was very low at r = .1608; with enjoyableness, it was also low at r = .2463. It was concluded that the learners did not spend more time on task because they perceived the materials to be more useful, or more enjoyable, but for some other reason.
The project also replicated part of a previous study. Green (1993) found a correlation averaging r = .6480 between the EFL learner-rated usefulness and enjoyableness of the activities in Puerto Rico. The present study also found the correlation high, at r = .7040.
Introduction: Rationale for the Study
This pilot study aimed at taking a preliminary step towards determining what keeps EFL learners on task in class; that is, to investigate with what factors increased levels of on-task behavior are associated. Two factors were investigated–the usefulness and enjoyableness of the materials in use on any one day. It was hypothesized before the study began that learners would spend more time on task when they perceived the materials in use to be useful and/or enjoyable. These hypotheses were tested by examining the correlation (strength of association) between levels of on-task behavior and the learner-perceived usefulness and enjoyableness of the materials in use over a whole term. The study also replicated part of Green’s 1993 study in Puerto Rico–the part that explored [-1-] the correlation between learner-perceived usefulness and enjoyableness.
(1) Correlating on-task behavior and usefulness; and (2) Correlating on-task behavior and enjoyableness.
Most (or all) EFL teachers would agree that maintaining a high level of on-task behavior (a combination of learner activity and persistence; see below) is both necessary and desirable within the classroom. There is some support for this contention: Crookes and Schmidt (1991, pp. 498-502) maintain that teachers are “convinced” that motivation as displayed in learner persistence with the task, activity, and concentration is crucial to second language success; and also that teachers are concerned not with the reasons for study, but with ensuring that learners are engaged in sustained study without continual encouragement (p. 480). Ushioda (1993, pp. 1-3) calls this a “practitioner-validated” view of motivation, and adds that learner participation, attentiveness, and enthusiasm can be regarded as significant outcomes in themselves, apart from the fact that they are “crucial” to L2 development. Schmidt (1990, p. 129) concludes that “paying attention is probably facilitative” to linguistic progress. Graves asserts (1993, p. 4) that most teachers regard learners’ being “active, eager and involved” as a measure of motivation, and defines motivation as a “continued desire to engage in learning.”
The following three assertions were made by Crookes and Schmidt in their influential 1991 article (pp. 486 and 492):
1. “Individuals will have a difficult time forcing themselves to attend for long to tasks that they perceive as irrelevant.”
2. “It seems likely that personal relevance can be [a] good determinant of selective attention.”
3. “A program which appears to meet the students’ own expressed needs … will be more motivating, more efficient, and thus more successful.”
Thus, the question investigated in the present pilot study is important to EFL teachers. Maintaining high levels of on-task behavior is both necessary and desirable; the existence of a consistent high correlation between on-task behavior and learner-perceived usefulness and enjoyableness has implications for the selection of materials for the EFL classroom. The premise that higher levels of learner on-task behavior positively affect learning outcomes can be accepted, in the absence of evidence to the contrary, as the default assumption. [-2-]
(3) Correlating usefulness and enjoyableness
The association between these two factors has received little attention, as Green (1993, p. 2) notes: “No previous studies known to me have investigated possible relationships between the reported enjoyableness and perceived effectiveness of different kinds of activities.” Green examined this relationship in his university-level EFL classroom in Puerto Rico. He reports (p. 8) that effectiveness and enjoyment ratings were highly correlated; the average correlation was r = .6480. In his conclusions (p. 8) he comments: “If the relationship between enjoyableness and effectiveness is … circular and mutually reinforcing, then teachers would be well advised to cultivate an enjoyable classroom environment.” He also calls (p. 8) for the study to be replicated with other populations in order to “identify potentially troublesome differences between teacher styles and student expectations.” As Green states, it is important that learner preferences for and opinions on EFL classroom materials are investigated, and that teachers do not rely solely on intuition and experience when selecting materials. My aim was to replicate this part of Green’s study in a South Korean setting.
Nunan’s definition (1992, p. 37) of on-task behavior was adopted for the present study: “students are ‘on-task’ when they are actively engaged in the pedagogic work of the day.” I also regard persistence (“concentrating attention or action on the same thing for an extended duration”; Crookes and Schmidt, 1991, p. 481) as part of on-task behavior, as I propose that learners who spend more time actively engaged in study are persisting with the set task.
Learners speaking to others in their group or to the teacher, or actively working within their assigned group, were regarded for the purposes of this study as being on task. The referent for on-task behavior was a score on an observation sheet designed to quantify on-task behavior. An observer recorded levels of on-task behavior.
It could be argued that it is not possible to observe on-task behavior because it takes place within the learner’s mind, and that a non-participant observer can hardly know whether a learner is on task or not. However, the inter-rater reliability of the on-task behavior observation sheet was computed at the high level of r = .9745 (p = .031) when two observers independently rated the on-task behavior of learners during a previous study in the same school (this will be further discussed below under “Collecting the Data.”) Both observers commented that they did not find it difficult–not as difficult as might be assumed–to assess and record levels of on-task behavior, and these comments are supported by the high [-3-] reliability computed for the instrument. This reliability is some indication (though not, of course, proof) that the behavior is observable. The inter-rater reliability of on-task behavior measures needs further empirical and theoretical checking in different settings and contexts.
“Usefulness” was defined as relevance to the goals of the learners in the target classes, effectiveness for language learning, and appropriacy. The referent was the combined score for four out of the ten items on a post-class questionnaire filled out daily by all learners in both classes. The questionnaire is found in the Appendix .
“Enjoyableness” was defined as that learner interest, enjoyment and enthusiasm that is generated by the materials in use. The referent was the combined score for six out of the ten items on the learner daily post-class questionnaire. One point to be considered regarding the concept of enjoyableness of materials is whether it has the same meaning in South Korea as in the West. Might it be possible that Korean learners do not regard enjoyableness of materials as desirable, or helpful? (The same question can, of course, be asked of all EFL learners around the world.) It could be argued that South Korean learners might not want to be “interested” in materials–that they might regard “interest” as an irrelevant or even undesirable feeling in EFL class; and also that they might accede to the teacher’s opinion on the materials. However, a certain degree of familiarity with Korean learners and culture (after eight years TEFL in South Korea) allows me to propose that South Korean learners want (as least as much as, for example, university-level learners of Spanish as a second language in the USA) to be given interesting and enjoyable materials, and also that they regard interesting and enjoyable materials as a desirable and necessary part of learning EFL. This is particularly true of the young adult learners in the present study. Some evidence that learners do not defer to the teacher was obtained in a previous study (Peacock, 1997, pp. 151-152) where learners in the same school frequently expressed negative as well as positive opinions about materials during post-class interviews. Deference to the teacher’s opinion was also reduced, of course, by having learners complete questionnaires anonymously in the present study. I have now come to regard the concept of enjoyableness of materials as applying almost equally in the West and in South Korea. The factor of potentially different responses to materials in differing cultures remains, however, an important one to consider before applying any data-collection instrument outside the culture in which it was first prepared and used.
Data Collection And Analysis
Two data-collection instruments, both quantitative, were utilized: [-4-]
(1) A tally sheet for quantifying learner on-task behavior
(2) A self-report questionnaire for learners
As no suitable instruments could be found in the literature, two existing instruments were extensively adapted:
(1) A tally sheet from Hopkins (1985, p. 95). The sheet used in the study consisted of 12 columns for scans and 18 lines for learners, and was designed to be filled out by a non-participant observer (or the class teacher) while learners were working in groups. The observer had to enter “1″ if learners were on task and “2″ if they were not on task for each scan, until all the learners had been observed 12 times.
(2) A questionnaire (from Gliksman, Gardner, & Smythe, 1982, p. 646) was very extensively revised and adapted for the study. It was filled out anonymously by individual learners. The aim of this written questionnaire was to let learners express their opinion on the usefulness and enjoyableness of the supplementary materials they had worked with on the day in question. The questionnaire was translated into Korean to avoid learner confusion; the English and Korean translation were given together. The questionnaire consisted of a list of ten items in opposition. Multiple items, chosen for their lexical similarity, were chosen for each variable in order to increase the reliability of the instrument. The items were: interesting/boring, relevant to my goals/not relevant to my goals, unenjoyable/enjoyable, meaningless/meaningful, appropriate for me/not appropriate for me, dull/exciting, satisfying/unsatisfying, useful for me/useless for me, unappealing/appealing, and absorbing/monotonous.
Items two, four, five, and eight (relevant, meaningful, appropriate, and useful), taken together, constituted the usefulness factor; and items one, three, six, seven, nine, and ten (interesting, enjoyable, exciting, satisfying, appealing, and absorbing) together made up the enjoyableness component. The appropriacy of dividing the ten items into these two distinct components was confirmed by factor analysis after the study.
Items were graded on a seven-point semantic differential scale. Learners put an “X” against the place on the scale that best described their opinion of the materials. For example, an “X” next to “boring” scored one on that item; an “X” next to “interesting” scored seven; and an “X” halfway between the two words scored four.
With the questionnaire, learners anonymously reported how useful and enjoyable they found the materials to be. The results from this instrument were perhaps less controversial than those from the on-task observation sheet, because learners were able to report their feelings directly. [-5-]
Two classes took part in the study, containing 22 learners in all (12 male, 10 female). The average age was 20, ranging from 18 to 23. All the learners were South Korean university students in a university EFL institute in Seoul, South Korea. Both classes were labeled “upper beginning.”
Collecting The Data
The research took place in two EFL classes assigned to this investigator for the 1995 summer term. The teaching materials used were typical of those adopted for South Korean university-level EFL. Among the authentic materials were two short poems, used in a listening and discussion exercise; television listings for that day from the local English-language newspaper, used for a short roleplay of choosing what to watch; three topical newspaper articles and the problem page, used for small group discussion and a related writing exercise; and a song on tape with a tapescript, for listening comprehension and as a stimulus for writing a dialog. Among the pedagogic materials used were a list of 20 personal questions, a list of controversial statements about marriage and family life, a list of questions designed to elicit learner opinions about their country, and a sheet on which learners were invited to make predictions about the future of South Korea and of the world. All the pedagogic materials were used for small group discussion.
On-task tally sheets and questionnaires were collected every day for seven weeks (over 20 separate days, as class was held four days a week, interrupted by holidays.) The stages of all activities were the same–the teacher introduced the activity, handed out the materials, went over the task, divided the class into groups of three, and started the activity. When the class had finished working and feedback had been conducted, learners filled out the post-activity questionnaires. A questionnaire was completed by each learner daily and then handed back to the teacher. A total of 40 tally sheets (20 from each class) and 391 questionnaires were collected over the term.
The inter-rater and intra-rater reliability of the on-task observation sheet had been computed in a previous study at r = .9124 (p = .031, N = 60) and r = .9745 (p < .001, N = 34) respectively (for details, see Peacock, 1997).
The internal consistency of the questionnaire was assessed after the present study via an alpha reliability check. The two components of the questionnaire, “usefulness” and “enjoyableness”, were checked separately. The result for “usefulness” was r = .8038 (p = .0161, N = 391), and for “enjoyableness” r = .9295 (p < .001, N = 391). I suggest that these results are an indication of a high level of internal consistency. [-6-]
The three-way correlations between on-task behavior and learner self-report of the factors “usefulness” and “enjoyableness” were checked via Pearson r. Conditions for the use of Pearson r were checked and were met.
(1) Results indicate a very low positive correlation between on-task behavior and learner report of usefulness; r = .1608 (p = .001, N = 391).
(2) Results indicate a low positive correlation between on-task behavior and learner report of enjoyableness; r = .2463 (p < .001, N = 391).
(3) Results indicate a high positive correlation between learner report of usefulness and enjoyableness; r = .7040 (p < .001, N = 391).
Discussion and Conclusions
(A) Results (1) and (2): little correlation between on-task behavior and the learner-rated usefulness and enjoyableness, of materials.
These results indicate that the learners in this study did not spend a higher proportion of the time on task because they perceived the materials to be more useful, or more enjoyable, but for some other reason. It should be noted that these findings run counter to the assertions made by Crookes and Schmidt (1991), reviewed above, that personal relevance will be a good determinant of selective learner attention.
The broad question raised in this study, of what factors affect levels of EFL learner activity in class, has not been answered even for the subject classes, except in a very limited way. However, I propose that the indications that (1) usefulness and (2) enjoyableness were not associated with increased levels of on-task behavior in the present study are of considerable value and interest to EFL teachers, as well as researchers, because they raise the question of what other factor or factors affect levels of on-task behavior in any EFL classroom.
It is noteworthy that class levels of on-task behavior varied considerably from day to day, and not just the levels of individual learners; that is, the classes tended to vary as a whole. I propose that it is possible to speculate that some or all the following other factors may be associated with increased or decreased levels of on-task behavior. The list of factors will be followed by separate subsections covering suggestions for EFL teachers, and for researchers: [-7-]
(1) The topic of the materials
(2) The type of activity undertaken by learners
(3) The teaching approach adopted by the teacher
(4) The motivation that brings learners to class–characterized by Gardner and Lambert (1972) as either “instrumental” or “integrative”
(5) Cultural factors
There are many factors that may affect levels of on-task behavior, either singly or in combination with each other; the possible existence of these unknown combinations may frustrate the search for the identification of individual factors. However, I suggest that further exploratory research such as the present study is needed in order to separate and identify these factors. If it is accepted that maintaining high levels of on-task behavior is both necessary and desirable, I propose that the factor or factors with which maintaining those levels might be associated deserves much further investigation within the EFL classroom.
Implications For Teaching EFL
The use of certain methods and materials will probably increase the levels of on-task behavior in any one EFL classroom to a degree noticeable to the teacher. If teachers are sensitive to the existence of changes in these levels, an ability to link increased on-task behavior with topic, activity type, or teaching method clearly has implications for planning further lessons with that class.
There may also be a link between extrinsic motivation, the use of particular materials, and on-task behavior. For example, with a class that was primarily integratively motivated, materials referring to the target language culture may increase on-task behavior. This also has implications for lesson planning.
Cultural factors may also play a role. Green’s study was in Puerto Rico and the present study in South Korea. I propose, though, that Korean learners want (as least as much as learners in the USA or Europe) to be given interesting and enjoyable materials, and regard them as desirable and as an important and necessary part of learning EFL.
I do not intend to generalize my findings outside the context of this study. The question raised, however–of what factors affect on-task behavior–should be examined in the local setting of any EFL teacher. The question of how far levels of on-task behavior vary from country to country was not, of course, investigated in the present study. The question deserves investigation, via cross-cultural comparison. [-8-]
I suggest that researchers hypothesize that increased levels of on-task behavior are associated with some or all the factors listed in the previous subsection, and with the learner-perceived usefulness and enjoyableness of materials. Additionally, the design of this study, which was an exploratory study, did not include the collection of further back-up data which might have illuminated and helped to explain the findings. If this study is replicated, I suggest that quantitative data from observations and questionnaires are backed up with data from post-class learner interviews, an end-of-term learner questionnaire, and perhaps learner diaries. The research also needs to be repeated in different settings, as proposed above, as well as with learners at different stages in their study of EFL. It is also very important that the assumed link (the “default assumption” referred to above) between on-task behavior and learning outcomes is examined. The difference between learner scores on pre- and post-term proficiency tests could perhaps be correlated with the other factors being investigated. For useful results to be obtained, though, data on proficiency would have to be collected over a much longer time than the seven weeks of the present study.
(B) Result (3): high correlation between the learner-rated usefulness and enjoyableness of materials.
This result is very similar to Green’s (1993, p. 6), who reports an average correlation of r = .6480 between the student-rated usefulness and enjoyableness of the activities in his EFL classroom; the correlation in the present study was r = .7040. I agree with Green’s proposal (1993, p. 8; reviewed above) that usefulness and enjoyableness are “mutually reinforcing,” and that therefore teachers would be well advised to cultivate an enjoyable classroom environment.
As a final conclusion, I propose that it is important to try to identify the factor or factors with which increased levels of on-task behavior are associated; the question of what these factors are has not yet been answered.
Crookes, G., & Schmidt, R. W. (1991). Motivation: reopening the research agenda. Language Learning, 41/4, 469-512.
Gardner, R. C. & Lambert, W. E. (1972). Attitudes and motivation in second-language learning. Rowley, MA: Newbury House. [-9-]
Gliksman, L., Gardner, R. C., & Smythe, P. C. (1982). The role of the integrative motive on students’ participation in the French classroom. Canadian Modern Language Review, 38, 625-647.
Graves, K. (1993). Japanese method. EFL Gazette, 165, 4.
Green, J. M. (1993). Student attitudes toward communicative and non-communicative activities: do enjoyment and effectiveness go together? Modern Language Journal, 77/1, 1-10.
Hopkins, D. (1985). A teacher’s guide to classroom research. Milton Keynes: Open University Press.
Nunan, D. (1992). Research methods in language learning. Cambridge: Cambridge University Press.
Peacock, M. (1997). The effect of authentic materials on the motivation of EFL learners. English Language Teaching Journal, 51/2, 144-156.
Schmidt, R. W. (1990). The role of consciousness in second language learning. Applied Linguistics, 11/2, 129-158.
Ushioda, E. (1993). Redefining motivation from the L2 learner’s point of view. Teanga, 13, 1-12.
About the author
Matthew Peacock teaches in the Department of English at the City University of Hong Kong. He taught EFL at university level in South Korea for eight years, before completing doctoral studies in the University of Essex. His current research interests include programme evaluation, materials evaluation, authentic materials, motivation, and learner attitudes and behavior.
© Copyright rests with authors. Please cite TESL-EJ appropriately.
Editor’s Note: Dashed numbers in square brackets indicate the end of each page in the paginated ASCII version of this article, which is the definitive edition. Please use these page numbers when citing this work.
AppendixSTUDENT QUESTIONNAIRE DATE: DD MM YY *Teaching Materials: *Do not write your name on this sheet - fill it out and give it back to your teacher. *The purpose of this questionnaire is to assess the value of the above teaching materials which were used in class today - NOT to assess the performance of you or you teacher. *This is not a test. There are no right or wrong answers; we want your own ideas and impressions. *Please mark ONE "X" on each scale to show how you rate the following concepts. Use the scales as follows:- *If the word at either end of the scale very strongly describes your ideas and impressions about the concept, you would place your check-mark as shown below: boring__X__:_____:_____:_____:_____:_____:_____interesting OR boring_____:_____:_____:_____:_____:_____:__X__interesting *If the word at either end of the scale describes somewhat your ideas and impressions about the concept (but not strongly so), you would place your check-mark as follows: boring_____:__X__:_____:_____:_____:_____:_____interesting OR boring_____:_____:_____:_____:_____:__X__:_____interesting *If the word at the end of the scale only slightly describes your ideas and impressions about the concept, you would place your check-mark as follows: boring_____:_____:__X__:_____:_____:_____:_____interesting OR [-11-] boring_____:_____:_____:_____:__X__:_____:_____interesting *Mark ONE "X" on each line: interesting_____:_____:_____:_____:_____:_____:_____boring relevant to my not relevant to goals_____:_____:_____:_____:_____:_____:_____my goals unenjoyable_____:_____:_____:____:_____:_____:____enjoyable meaningless_____:_____:____:____:_____:_____:____meaningful appropriate not appropriate for me_____:_____:_____:_____:_____:_____:_____for me dull_____:_____:_____:_____:_____:_____:_____exciting satisfying_____:____:_____:_____:____:____:____unsatisfying useful for me____:___:___:___:____:____:____useless for me unappealing_____:____:_____:_____:_____:____:_____appealing absorbing_____:____:_____:_____:_____:_____:_____monotonous [-12-]