November 2020 – Volume 24, Number 3
Emrah Cinkara
Gaziantep University
<cinkaragantep.edu.tr>
Abstract
Focusing on the reflective exchanges of four test writers at an intensive language school within a public university, this study explored the effectiveness of a “lesson study” model on their professional development as test writers. Lesson study is a powerful tool for teachers’ professional development and a modified version of it—a quiz study—was used in the process of developing, administering, reviewing the results and evaluating feedback of the test. Participants were audio recorded throughout the four stages of the quiz study cycle as they discussed a variety of topics. Moreover, content analysis of this data revealed that the test writers had negotiated two themes: assessment literacy and content-specific knowledge and skills. The significance of this study lies in understanding the nature of the quiz study process as well as test writers’ professional development areas, which may provide valuable insight for all test writers, teachers, and teacher educators.
Keywords: EFL testing; lesson study; reflection; professional development; test writers
Introduction
Quiz Study as a Professional Development Model
The Organisation for Economic Co-Operation and Development (OECD) published a report on the professional development of teachers from 23 countries and determined the specific needs for teachers’ professional development areas. In this report, around 20 percent of the participants cited assessment practices as a high development-need area (2009, p. 60). In a local context, a needs analysis study to improve the quality of the Thailand National Test revealed the highest area in need of improvement is the professional development of test writers. (Sapsombat & Roengsumran, 2019). To the best of this researcher’s knowledge, there is no such study investigating lesson study among test writers. Therefore, the researcher will take a more general stance and examine lesson study within a teachers’ professional development perspective. The professional development of teachers commences with their initial teacher education and continues throughout their professional lives. For some researchers, professional development includes early learning experiences prior to pre-service education (Attia, 2014). In a general sense, all teaching and learning activities along with pre-service and in-service training play a significant role in teachers’ professional development in terms of improving teaching practices as well as shaping their construction of professional identities (Klenowski, Askew & Carnell 2006).
There are a wide range of context-specific teacher development models around the world and these are reflected in teacher training practices. Teachers’ development activities within these various models extend from pre-service Bachelor’s-level education to attendance at workshops and professional meetings as well as informal undertakings following field-specific publications (Jones & Dexter 2014). Other development activities may include in-service training and short-term courses which present new advancements in teachers’ fields. These, however, generally ignore participants’ individual needs and contexts.
The present study regards professional development as the result of a teacher’s gaining increased experience and examining their teaching systematically, which may be formally planned and implemented on a large scale or occur informally in the form of school-based activities (Glatthorn, 1995, p. 41). This perspective has inspired numerous professional development models, one of which is the lesson study model. The lesson study model was used in this article as a professional development activity for test writers at a tertiary level intensive English program. In lesson study, teachers usually work together to plan, implement, and review a lesson, while here, we will examine test writers who worked together to plan, administer, and review a quiz. Thus this study model here in is referred to as quiz study. The modifications in the quiz study model include a) designing and administration of a quiz rather than a lesson plan, b) including test writers, coordinators, and teachers as participants in the study as opposed to only teachers, and c) getting feedback from test writers, coordinators, teachers, and students as opposed to solely teachers and students in lesson study. To the best of the author’s knowledge, the lesson study model has not been employed in the test development process so far, so the framework is taken from its uses in teaching subject matter contexts. The following section will review the theoretical background and related literature for lesson study.
Lesson Study
The lesson study model refers to the process of making preliminary preparations that enable teachers, prospective teachers, and other specialists to work together in a teaching context in the most effective manner as well as to evaluate and implement this practice and its outputs (Lewis, 2002). This model first emerged in Japan and was named after the words in-house (konai) and training (kenshu) (Fernandez & Yoshida, 2004). One notable factor that has led to the internationalisation of this original model is the fact that it enables teachers to research the semantic structures of students and provides them with opportunities for acquiring conceptual knowledge while devising lesson plans (Parks, 2008).
The model reinforces reflection skills—which are vital to professional development—by means of discussion-based knowledge shared both individually and on a group level (Watanabe, 2003). The lesson study model has been described as supporting teachers in terms of general, professional, and content knowledge; knowledge of students’ conceptions; and practice involving lesson planning, conducting the plan, observing peers, and reflecting on peer- as well as self-performances (Fernandez &Yoshida 2004; Lewis, 2002). In addition to benefiting teachers personally in terms of professional development, the lesson study model has also been proven to benefit institutions themselves. For example, it has been shown that teachers who participate in lesson study activities initiate positive institutional change by contributing to innovation and learning cultures in their school settings (Sitton, 2006).
Lesson study is a cooperative professional development model that focuses on research and implementation (Watanabe, 2003). The model works in a cyclical manner: in the first step, teachers come together and plan the lesson; in the second, one of the teachers facilitates the planned lesson; and finally, both teachers evaluate the effectiveness of the implemented lesson plan. Even though these three steps constitute the core activities of the lesson study model, additional activities such as preliminary meetings, utilization of scientific studies, and pre- and post-implementation focus group meetings can be included in the process (Murata & Pothen, 2011). The aims of lesson study and its additional activities have been described by Lewis (2002, p. 12) as follows:
- setting goals for students’ learning and long-term development,
- planning research lessons in a cooperative manner in order to achieve these goals,
- implementing research lessons in which one teacher implements and the others observe students’ learning and development processes,
- discussing the information gathered during implementation and conducting evaluations in order to improve the education within a more generalized scope.
One of the most important outcomes of lesson study is new and innovative learning resources developed during the implementation of the lesson study (Lewis, Perry & Murata, 2006). Teachers’ previous understandings of teaching as well as other aspects of their professional knowledge have been investigated in relation to professional development and lesson study is reported to be directly related with teachers’ self-efficacy, which is clearly linked with success in professional development activities (Sibbald, 2009). Sitton (2006) has also reported that lesson study is perceived by teachers as a satisfying professional development program. Similarly, lesson study has been described as an effective method in terms of student-centred pedagogical skills and the development of mathematics content/professional knowledge among teachers (Bell & Gilbert, 2004; Elipane, 2012; Lewis, 2002; Watanabe, 2003).
Teachers experience two main benefits of lesson study: (1) collaboration in lesson study prevents teachers from feeling isolated in terms of their profession, and (2) teachers focus more on the learning process of students, thus facilitating more student-centred lessons (Cajklera et al., 2015). Additionally, teachers better comprehend the significance of communication with students regarding their experiences in comprehending the derivative concepts (Verhoef et al., 2015). Moreover, it has been reported that pre-service teachers, in-service teachers, and students all benefit greatly from this practice (Cajklera et al., 2015; Cerbin & Kopp, 2006; Verhoef et al., 2015).
The possible relationship between lesson study and the pedagogical content knowledge of teachers has been widely investigated (Cerbin & Kopp, 2006; Coenders & Verhoef, 2019). Pedagogical content knowledge was first mentioned by Shulman (1986) and it expresses that knowing about a topic is not enough for teaching it. According to Shulman, pedagogical content knowledge is the determination of elements that can either ease the learning of certain subjects or complicate them. It is an investigation of the most effective way for students—who have different profiles and, thus, different needs—to learn the subject at hand (Shulman, 1986). By utilizing lesson study, Cerbin and Kopp (2006) suggested a model that aims to improve the pedagogical content knowledge of teachers and, thus, improve teaching practices.
In addition to in-service teacher development, studies have also reported the effects of lesson study on student success (Elliott & Yu, 2013) and pre-service teacher education (Yalcin Arslan, 2019). Additionally, it was reported that the cyclic continuum of lesson study practices had positively affected students on various success levels.
Objectives of the Present Study
It is clear that presently there is a gap between pre-service and in-service teacher education/training and needs of teachers – especially test writing teachers – in terms of professional development regarding assessment (OECD, 2009; Popham, 2011; Sapsombat & Roengsumran, 2019). As the literature review shows, the lesson study might present opportunities for test writers to reflect on their practice and improve their test writing skills with collaborative work. Within a socio-cultural theory framework, this study regards teacher development as a collective process of sharing, developing, and influencing each other. It is actually a challenging task to mobilise this knowledge across teachers and classrooms (Edwards, 2014). With this understanding, this study aims to identify and explain professional development activities throughout a lesson study process, called quiz study in this study, among a group of four EFL test writers. For this purpose, the current study aims to answer the following research questions:
- Does a quiz study model provide sufficient reflection and development opportunities for language-test writers?
- What insights into language-test writers’ professional development emerge from their reflections during the four stages of three quiz study cycles?
Method
Research Design
This study adopted a qualitative, exploratory, and inductive research design. A case study approach was employed during the Fall 2017 Academic Semester. This approach commonly is used to generate an in-depth, multi-faceted understanding of a complex issue in its real-life context (Crowe et al., 2011, p. 1). This study observed professional development activities within a quiz study model completed in three full cycles by EFL test writers. Each cycle lasted for two weeks and data were collected over an eight-week period. The investigations focused on EFL test writers’ reflections on the test construction process, feedback from teachers who administered the tests, interpretations of the items they wrote on three quizzes, and whether or how these activities supported test writers in terms of their professional development.
Research Context
Data were collected within an intensive English preparatory program at a public university in Turkey in 2018/19 academic year. The school caters to the needs of 1,450 students who must successfully complete the program to continue with their intended BA/BSc degree programs delivered in English. The preparatory program extends over the course of four modules levelled A1, A2, B1, and B2, progressively. Students are placed at an appropriate level based on the results of a placement test and each module lasts for eight weeks. Assessment tools in each model include five quizzes, one end-of-module exam, participation, and online tasks with different weights, 20%, 65%, 5%, and 10%, respectively. Students with at least 60% success may continue to the next level, but unsuccessful students must repeat the same level. There are ninety-one instructors and twenty-four hours of lessons every week. Students are placed in seventy-four groups and the assessment policy is the same for all groups in the same module.
The curriculum, course materials, and exam dates within each level correspond with those of other levels. For this reason, the school has a testing office that creates and organizes all formal assessment tools, including quizzes, portfolio tasks, end-of-module exit tests, and proficiency tests. Four instructors with at least five years of teaching experience in the institution work in the testing office without any teaching load. Before commencing their work in the office, they participate in an orientation pertaining to assessment-related issues and office operations.
Research Participants
There were two groups of participants in this study: four testing office members (test writers) and seven teachers who administered the tests created by these members. As the study aimed to discern the effects of the quiz study model on test writers’ professional development, only the data of the test writers were included in the study. Of the four test writers, three were female and one was male, all of them having at least two years of experience in the office. Ayla and Sinem held MA degrees while Kutay and Ceren held BA degrees in English language-related fields (pseudonyms are used here to ensure confidentiality). Ayla, Sinem, Kutay, and Ceren had 17, 16, 9, and 8 years of English language-teaching experience and 4, 3, 2 and 2 years of testing office experience, respectively. They all possessed assessment knowledge from their BA-level courses and teaching experience.
The test writers indicated their willingness and signed a written consent to participate in professional development activity for the purposes of this study. All voluntarily participated in at least three in-house professional development activities over the previous two years and Ayla, who was also enrolled in PhD courses at the time, participated in two academic conferences during the previous year.
Quiz Study Process
As the main focus of this research was to investigate language-test writers’ reflections on their planning, development, administration, and review processes of a quiz, the study model in this research is referred to as quiz study. Quiz study as a professional development model has been modified from Dudley (2011)’s lesson study model. The modifications are illustrated in Table 1:
Table 1. Comparison of Quiz Study and Lesson Study.
Model | Task | Participants | Feedback sources |
Quiz study | writing and administering a quiz | test writers, coordinators, and teachers | test writers, coordinators, teachers, and students |
Lesson study | writing and teaching a lesson plan | teachers | teachers and students |
As Table 1 shows, the modifications in the quiz study model included a) designing and administration of a quiz rather than a lesson plan, b) including test writers, coordinators, and teachers as participants in the study as opposed to teachers, and c) getting feedback from test writers, coordinators, teachers, and students as opposed to teachers and students in lesson study. The quiz study process used in this study is illustrated in Figure 1:
Figure 1. Quiz Study Cycles.
The study started with a first meeting where a general outline of the research was shared and quiz study was discussed with participants. Then, the quiz study cycle began in the second meeting with teachers’ planning of the upcoming exam for around 1200 students. In the third meeting, the test writers reviewed the items they individually created and constructed a final version of the quiz. The quiz was then administered and test writers met for the fourth time to get feedback from teachers who administered the quiz to their students.
The quiz study was completed in three cycles, each consisting of four stages. Before the first cycle, an initial meeting was conducted to inform participants about the study and its procedures. Participants in this meeting agreed to adhere the following responsibilities:
- to share information on their quiz development, administration and evaluation processes,
- to manage meetings with teachers about the development, administration, and evaluation processes,
- to reflect on the processes and sharing teachers’ feedback,
- to participate in post-study interviews with the researcher.
In the first meeting, the test writers decided on the content and specifications of the quiz. Then, in the second step, the quiz created by the test writer was reviewed by the other test writers and teachers. In the third step, the quiz was administered to students during class time, and in the last step, teachers shared their feedback and students’ feedback on the quiz with the test writers. This procedure was repeated three times as illustrated in Figure 1.
Data Collection and Analysis Procedure
The data for this study was collected throughout the quiz study cycle presented in Figure 1. A total of 12 meetings and one interview with each participant after the study were audio recorded, each of which lasted from 18 to 42 minutes. The audio recordings (total of 248 minutes of quiz study cycle meetings and 44 minutes of individual post-interviews) were transcribed verbatim and qualitative data was inductively analysed. The researcher utilised grounded theory principles to extract themes from the data (Fraenkel, Wallen, & Hyun, 2015), in order to find out the ways that the quiz study model improved the participants’ proficiency in terms of their test writing practices. For the purpose of trustworthiness, the data from quiz study meetings and interviews were analysed by two independent coders—the researcher as coder one and a colleague holding a PhD in English Language Teaching as coder two. First, the study and data was introduced to coder two and the transcripts were carefully read by the coders independently for professional development-related utterances and the detected examples were annotated. Then, the examples were coded based on their content. Finally, a discussion session was conducted by the two coders to improve the rigor of the analysis and only the mutually agreed-upon utterances and codes were included.
Findings
The participants shared their comments, reflections, and evaluations related to the various stages of the test development and administration process. An analysis of the qualitative data revealed that participants had negotiated and improved themselves on two testing-related themes: assessment literacy and content knowledge and skills. The findings, along with example extracts, are presented in the following section. All the names are pseudonyms and the steps in which the extracts were taken are given in parentheses after each extract.
Domain of assessment literacy
Assessment literacy is defined as familiarity with knowledge and skills about the principles, methodology, techniques, and specific terminology of assessment (Popham, 2009). Test writers in this study learned from and taught each other about the mechanics of writing a test, including but not limited to setting, scheduling, timing, presentation, format, response as well as validity, reliability, and practicality. For instance, in the following extract, two test writers discuss the timing to be allocated for the quiz:
Excerpt 1 (Cycle 2: Reviewing the quiz)
Kutay: Do you think one lesson will be enough?
Sinem: Why?
Kutay: It takes more than that.
Sinem: I don’t know. What do you mean?
Kutay: Um, the questions…too many questions and the students are not that quick.
Sinem: Oh, you mean the time allocated for the quiz.
Kutay: Yes, it will be too short.
Sinem: Well, we generally give one minute for each question and 5 to 10 minutes for reading and listening if available. This quiz has 42 questions with one short reading text.
Kutay: Oh, nice! I like the way you calculate it [time allocated for the quiz].
In Excerpt 1, after reviewing the quiz written by Sinem, Kutay clearly expressed some doubts about the appropriate time that should be given to students. He shared this concern with Sinem, who explained how she had calculated the time of the quiz. Through this conversation, Kutay learnt about something which could only be gained through experience in a specific context. Test writers also discussed about validity related issues. For example, in the following excerpt, Ceren and Sinem talked about how a test task did not match with the language function addressed in the item:
Excerpt 2 (Cycle 1: Reviewing the quiz)
Ceren: These situational matching items are a little bit odd for me. I mean, we are assessing functional aspects but we utilize matching tasks.
Sinem: Okay, they need to match them, that’s easy.
Ceren: I am not worried about it, but it should be more productive rather than selective from a list of options.
Sinem: I see. Maybe I can turn it into a dialogue completion task.
Ceren: Yes, that would be much better.
Excerpt 2 shows that Ceren had concerns about what the task required in the item. She felt that the linguistic element addressed in the matching item should be assessed in a more productive manner. Matching items only require students to recognize language elements given in the option bank; however, dialogue completion calls for recall in which students have to remember the linguistic element to provide the answer without and clues given in the item. Similarly, in the following excerpt, Sinem and Ayşe discuss a reliability issue with a cloze test item:
Excerpt 3 (Cycle 3: Reviewing the quiz)
Sinem: Um, this cloze text item…
Ayşe: Yep?
Sinem: There are the alternatives above it and the number of alternatives is the same.
Ayşe: Hmm, the same?
Sinem: The number of items and blanks are the same.
Ayşe: What about it?
Sinem: You know it is better to have extra. When a student has an incorrect answer, it automatically produces another incorrect answer.
Ayşe: Hmm, I see.
In Excerpt 3, Sinem and Ayşe discussed a problem regarding the reliability of the quiz results, which emerged as a result of giving the same number of alternatives and blanks on a banked cloze-test item. Normally, banked cloze tests should have extra items as distractors; however, Ayşe did not include any distractors in the bank. Noticing this, Sinem was worried that it would result in reliability issues with the quiz. Then, they agreed on the issue. This exchange of knowledge and experience is considerably significant to comprehend and practice on the essence of what actually constitutes a test’s reliability rather than simply calculating a test reliability coefficient (Popham, 2011, p. 267). In another excerpt, Ceren and the teacher giving feedback on her quiz discuss a practicality issue:
Excerpt 4 (Cycle 1: Feedback from teacher and students)
Teacher 2: Hocam [an addressing word meaning teacher], why do we have to carry these CDs [CD players]?
Ceren: For the listening part, of course.
Teacher 2: No, I know, but we have computers and the internet in each class. Why don’t we just put them online and email the link to proctors?
Ceren: Yeah, it might be a good idea, but we have old teachers who may have problems with this. Plus, what if the internet is cut off?
Teacher 2: Then, we carry the CD players.
The teachers proctoring exams must carry their CD players with them to the test rooms in order to play a listening file. However, Teacher 2 offers an alternative to this and suggests a more practical way of administering the listening part of the quiz. It is a quite simple suggestion: instead of burning audio files onto CDs and making proctors carry their CD players, sharing the audio file online with the proctors would be more efficient. However, Ceren states her concern about this by presenting some issues with proctors lacking technical skills and possible internet problems.
Also participants’ statements in the interview showed that they benefited from the quiz study model to better design their test development procedure. The institution where the study was based has a checklist for test development. However, after this study, participants requested some revisions on the checklist and they created a comprehensive flowchart for test writing:
Excerpt 5 (Post interview with Ceren)
Ceren: … I was not very enthusiastic about this this activity (quiz study) at the beginning.
Interviewer: Oh, yeah?
Ceren: Yeah, ummm, after like the second quiz, I thought that this is not a simple academic research, we (the testing office) decided to revise our checklist. ın this research we did not only work at the theoretical level, we also put into practice what we know.
Interviewer: What do you mean?
Ceren: For example, we thought we have to (get) feedback from teachers as well as our coordinators. Then, we should record the feedback on each item in the test for future use. So, the office changed the checklist and included teachers and students.
Domain of content knowledge and skills
Another point emerging in the qualitative data was that test writers and teachers discussed subject-specific content knowledge and skills, i.e. grammar structures, vocabulary, type of reading and listening texts, etc. Test writers design exams based on course content and predetermined objectives. Aware of this, test writers plan their quizzes based on the content detailed by the teaching program. Accordingly, content analysis of the qualitative data demonstrated that test writers had improved their English language knowledge and skills while designing the quizzes. This is clearly illustrated by Sinem in the post-interview:
Excerpt 6 (Post interview with Sinem)
Sinem: …Throughout this study, I learnt a lot from other testers and teachers as well. For instance in the last quiz, one of the questions I wrote for the reading text was incorrect. In the study (Cycle 3-Reviewing the quiz), Ceren told me about incorrect use of ‘them’ pronoun. I was quite sure that it was a correct use, but then I learnt that it had to be ‘themselves’.
Ceren in Excerpt 6 expresses the way they improved their teaching/assessment content knowledge, which is knowledge of reflexive pronouns. For instance, in the following excerpt, Cansu and Sinem discuss a specific word used in the quiz:
Excerpt 7 (Cycle 2: Reviewing the quiz)
Cansu: I have a few concerns about this part, especially the words here.
Sinem: Which ones?
Cansu: Orchard and shovel. They [the students] haven’t seen these words.
Sinem: But they can understand it [the text] without knowing all the words.
Cansu: Yeah, I know, but it is not fair to use an unseen word.
Sinem: Okay. I will just say garden, then.
Cansu is a teacher in the module for which Sinem is responsible. She states that the vocabulary addressed in the discussed item has not been seen in class and, therefore, should be modified. Then, based on this feedback, Sinem decides to change the item so that it does not include those words. Similarly, a test writer and a teacher discuss below the difficulty level of a listening task:
Excerpt 8 (Cycle 2: Reviewing the quiz)
Aziz: What do you think of the listening part? I thought it was a bit too difficult for this level [A2 level].
Ceren: Really?
Aziz: Yes, it is difficult, you know, and the accent is a heavy British accent. My students won’t understand anything from this. Trust me.
Ceren: Okay. I will discuss it with Ayla [A2-level coordinator] and try to change it.
In the above excerpt, Ceren, the A2 test writer, receives comments from Aziz, who has just reviewed the quiz written by Ceren. Aziz complains about the specific accent used in the listening task and Ceren agrees to discuss the listening text with the A2-level coordinator. In steps 2 and 4, feedback from teachers and students include some fruitful discussions on similar issues. In another extract, Aylin and Kutay discuss the answer key and an alternative answer:
Excerpt 9 (Cycle 1: Feedback from teacher and students)
Aylin: Hocam, for Question 13, one of my students had an alternative. Can we also accept “she didn’t want to go” as a correct answer?
Kutay: Um, let me see…This one, I don’t quite agree. Here it says, “I was not feeling very well”. Here it says, “I didn’t want to go to Jolie’s party”. Hocam, it is clear that she didn’t go to the party because she was sick.
Aylin: But, it also says I didn’t want to go. We should also accept it.
The above excerpt indicates that, based on her student’s feedback, Aylin thinks that there should be an alternative correct answer to an item on the quiz. They discuss the alternative from the reading text and are unable to reach an agreement. Therefore, they decide to discuss this issue with the level coordinator. This was later discussed with all test writers and coordinators, and the final resolution (accepting the alternative answer) was shared with all teachers via institutional group email.
Discussion
This study evaluated the proposed quiz study model as a professional development activity model among language-test writers by analysing test writers’ reflections throughout the four stages of three quiz study cycles and individual post-interviews. Overall, the results of the study revealed that the quiz study may be used as a professional development tool for test writers in similar institutions to improve test writers’ assessment literacy skills and content knowledge and skills. Especially preparatory and intensive language programs in Turkey and around the world mostly have their own test writing team to create very frequent tests some of which are high stakes and gate keeping. Therefore, professional development of the test writers is of great significance for the reliability, validity and trustworthiness of these tests and institutions.
Regarding the first objective, the data revealed that the quiz study model had provided test writers with opportunities to interact with their peers and with teachers who commented on the draft of the quiz, administer the final version, and give feedback on the results (both theirs and their students’). This finding corresponds with research on the original lesson study model, which reported that lesson study had improved teacher’s engagement with professional learning, reflection, and classroom-based research (Edwards, 2014). Peer observation is a potent professional development tool for teachers which provides valuable insights into effective educational practices (Drew et.al., 2017). Similarly, in this study, test writers could observe their peers’ test development processes and share their comments about various aspects of these processes. The findings of the current study strongly suggest that the quiz study model had provided peer observation opportunities for test writers in which they could observe, reflect, and learn. Participants benefited from these experiences, which included many reflections on experiences regarding the aspects of test development and subject-specific content knowledge related to the English language. Reflection during professional development activities enable individuals to analyse critically their experiences and learn from them (Benade, 2015; Cinkara, 2016; Richards & Lockhart, 1994).
Within a social constructivist perspective, knowledge in general and teacher identity as well as teacher knowledge are socially constructed phenomena (Adams, 2006), and in the quiz study process participants generated and shared knowledge into assessment literacy as well as content knowledge and skills. As opposed to some traditional professional development activities such as seminars, short courses, etc., the quiz study model provides opportunities for test writers to come together, plan the quiz, review the quiz in group, administer it and obtain feedback from their peers and students. Therefore, the lesson study model brings the social-constructivist aspect into play in test writers’ professional development and helps them to obtain constant feedback from the group (Chokshi & Fernandez, 2005). Therefore, this model emerges as an alternative for traditional individualistic professional development activities.
Pertaining to the second objective of the study, emerging insights into language-test writers’ professional development from the test writers’ reflections were discovered and the themes were assessment literacy and content knowledge. Participants in this study reportedly held basic assessment literacy from assessment courses in their pre-service teacher education institutions and from their personal readings. Our findings indicated that study participants had commented on issues such as reliability, validity, practicality, time allocation, scoring, etc. Possessing the knowledge and skills regarding assessment improves test writers’ ability to assess learners’ knowledge and abilities as well as learners’ awareness of their achievement (Scarino, 2017). This literacy enables test designers to determine context-appropriate assessment tools and use the best alternative (Popham, 2009). It is quite apparent that assessment literacy is among the key components of successful assessment practice, and the quiz study model in this study significantly contributed to the basic understanding as well as practical aspects of assessment literacy.
The second theme emerging from the data concerned content knowledge and skills. As foreign language speakers of English, the participants discussed subject-specific issues related to the English language. They were not native speakers, but they were able to seek advice from their native English-speaking peer when constructing the test items. However, as previously stated in the excerpts, there may be some disagreements pertaining to the appropriate use of language in the instructions, texts, items, and answers. While creating language tests, many subject matter-specific issues such as proficiency levels, text readability, and type of English used, pace of the listening text, etc. play a significant role in the effective administration of the test and learner achievement (Hakuta, 2009). Our participants discussed the content of the quiz as well as the alternatives to the answer key and made clear references to this issue, which demonstrates the efficiency of the proposed quiz study model in test writers’ professional development.
The present findings extend to the reflective nature of assessment and quiz study. One of the main objectives of assessment is to provide feedback to learners, which is an important tool for improvement in education (Hyland, 1990). Similarly, feedback provides insights for teaching and assessment practices. In a regular educational setting, teachers or test writers are expected to write their tests, administer them, and obtain student feedback. However, with quiz study, teachers and tests writers can work together while planning, writing, reviewing, administering, and evaluating the results. In all of these stages, they can obtain constant and immediate feedback from their peers as well as students taking the quiz. Therefore, the quiz study model is an effective professional development activity.
Testing is an important component of teaching and learning as it not only provides insights into the efficiency of teaching but also serves to assign grades, improve performance, enhance retention, diagnose weaknesses, and motivate students (Roediger, Putnam, & Smith, 2011). This aspect of teaching should be systematically developed in teachers through pre-service and in-service training. Moreover, there is a great demand by teachers for professional development activities to address their needs, as stated in the OECD report (2009). This study presented a lesson study model in which test writers could plan, design, write, administer, and reflect on three cycles of quiz construction. The analysis of participant comments, reflections, and exchanges indicated that test writers had improved their assessment practices via quiz study in two domains: assessment literacy and content knowledge and skills. Also, these sessions provided opportunities for test writers to reflect on their test construction processes and seek expertise in terms of subject matter and assessment literacy.
Professional development in teaching is an indispensable and lifelong activity which requires clear and appropriate connections to teachers’ specific work and contexts. Moreover, it is important to incorporate professional development activities in contexts in which the gained skills and knowledge may be utilised (National Research Council, 1996). In this study, employing the quiz study model in a foreign language context presented positive outcomes with respect to participants’ specific needs, reflections, and questions regarding the assessment tools they had designed. In various stages of the quiz study cycle, they sought the knowledge and expertise of their peers in terms of subject matter and assessment literacy. While professional development activities which do not originate in participants’ contexts are inherently ineffective (Klenowski, Askew & Carnell, 2006; OECD, 2009), this model is productive as it originates in participants’ contexts and therefore relates to their own objectives and problems.
Conclusion
Assessment is a key issue in education and teachers should be supported by professional development activities related to it. This need has officially been stated by institutions such as the OECD (2009, p. 60) and research (Popham, 2011). However, few models have been proposed in this regard. Therefore, the current study proposes that the quiz study model might be an effective tool for meeting professional development needs in terms of assessment. It suggests that the quiz study model could be implemented in similar teaching contexts as well as in teacher education contexts in which teachers may introduce the quiz study model to pre-service teachers as an alternative professional development model.
The study is not free of constraints or limitations. To begin with, the data collection was limited to three quiz study cycles and one instrument over a limited time period. This time period could be extended and more study cycles could be implemented in future research. Secondly, the main assumption of the study model is to collaborate and learn from each other in test writing process. This may not be achieved in contexts where teachers’ assessment knowledge and skills are rather limited.
About the Author
Emrah Cinkara is an associate professor at the English Language Teaching program at Gaziantep University, Turkey. His research interests include but not limited to professional development of teachers and test writers, test writing processes, teacher identity development, and transnational teacher identity. He is currently the Director of the School of Foreign Languages at Gaziantep University.
References
Adams, P. (2006). Exploring social constructivism: theories and practicalities. Education, 34(3), 243–257. https://doi.org/10.1080/03004270600898893.
Attia, M. (2014). The role of early learning experience in shaping teacher cognition and technology use. In P. Breen, (Ed.), Cases on teacher identity, diversity, and cognition in higher education (pp. 1–21). Hershey, PA: IGI Global. https://www.igi-global.com/gateway/chapter/107871.
Bell, B. & Gilbert, J. (2004). A model for achieving teacher development. In J. Gilbert, (Ed.), The Routledge Falmer reader in science education (pp. 258–278). New York: Routledge Falmer.
Benade, L. (2015). Teachers’ critical reflective practice in the context of twenty-first century learning. Open Review of Educational Research, 2(1), 42–54. https://doi.org/10.1080/23265507.2014.998159.
Cajklera, W., Wooda, P., Nortona, J., Peddera, D., & Xua, H. (2015). Teacher perspectives about lesson study in secondary school departments: A collaborative vehicle for professional learning and practice development. Research Papers in Education, 30(2), 192-213. https://doi.org/10.1080/02671522.2014.887139.
Cerbin, W. & Kopp, B. (2006). Lesson study as a model for building pedagogical knowledge and improving teaching. International Journal of Teaching and Learning in Higher Education, 18(3), 250-257.
Chokshi, S. & Fernandez, C. (2005). Reaping the systemic benefits of lesson study. Phi Delta Kappan, 86(9), 674–80.
Cinkara, E. (2016). Reflective practice and foreign language classroom anxiety: ideo-stimulated recall at work. Reflective Practice, 17(6), 694-707. https://doi.org/10.1080/14623943.2016.1206880
Coenders, F. & Verhoef, N. (2019). Lesson Study: professional development (PD) for beginning and experienced teachers. Professional Development in Education, 45(2), 217–230. https://doi.org/10.1080/19415257.2018.1430050
Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case study approach. BMC Medical Research Methodology, 11(1), 100-112. https://doi.org/10.1186/1471-2288-11-100.
Drew, S. L., Phelan, K. Lindsay, A. Carbone, B. Ross, K. Wood, S. S., & Cottman, C. (2017). Formative observation of teaching: Focusing peer assistance on teachers’ developmental goals. Assessment and Evaluation in Higher Education, 42(6), 914–929. http://dx.doi.org/10.1080/02602938.2016.1209733.
Dudley, P. (2011). Lesson study: A handbook. Www.Lessonstudy.Co.Uk.
Edwards, S. G. (2014). Lesson study: A mechanism to support effective teacher engagement with and in educational research? A think-piece. Journal of Educational Research, 1(1), 48–64.
Elipane, L. E. (2012). Integrating the elements of lesson study in pre-service mathematics teacher education. 12th International Congress on Mathematical Education, 8 July – 15 July, 2012, COEX, Seoul, Korea
Elliott, J. & Yu, C. (2013). Learning studies in Hong Kong schools: A summary evaluation report on the ‘variation for the improvement of teaching and learning’ (VITAL) project. Education & Didactique, 7(2), 147–163.
Fernandez, C. & Yoshida, M. (2004). Lesson study: A case of a Japanese approach to improving instruction through school-based teacher development. Mahwah: Lawrence Erlbaum.
Fraenkel, J.R., Wallen N.E., & Hyun H.H.. (2015). How to design and evaluate research in education. New York, NY: McGraw-Hill.
Glatthorn, A. (1995). Teacher Development? In L. Anderson (Ed.), International encyclopaedia of teaching and teacher education (2nd Edition). London: Pergamon Press.
Hakuta, K. (2009). Guidelines for the assessment of English language learners. Educational Testing Service. https://www.ets.org/s/about/pdf/ell_guidelines.pdf.
Hyland, K. (1990). Providing productive feedback. ELT Journal, 44(4), 279–285.
Jones, W.M. & S. Dexter. (2014). How teachers learn: the roles of formal, informal, and independent learning. Educational Technology Research and Development, 62(3), 367–384. http://link.springer.com/10.1007/s11423-014-9337-6.
Klenowski, V., Askew, S., & Carnell E.. (2006). Portfolios for learning, assessment and professional development in higher education. Assessment & Evaluation in Higher Education, 31(3), 267–286. https://doi.org/10.1080/02602930500352816.
Lewis, C. (2002). Lesson study: A handbook of teacherled instructional change. Philadelphia: Research for Better Schools.
Lewis, C., Perry, R., & Murata, A. (2006). How should research contribute to instructional improvement? The case of lesson study. Educational Researcher, 35(3), 3–14. https://doi.org/10.3102/0013189X035003003
Murata, A. (2011). Introduction: Conceptual overview of lesson study. In L. C. Hart, A. Alston and A. Murata (Eds.), Lesson study research and practice in mathematics education, (pp. 1-12). Dordrecht, the Netherlands: Springer.
Murata, A., Pothen, B. E. (2011). Lesson study in preservice elementary mathematics methods course. In L. C. Hart, A. Alston and A. Murata (Eds.), Lesson study research and practice in mathematics education, (pp. 103-116). Dordrecht, the Netherlands: Springer.
National Research Council. (1996). National science education standards. Washington, D.C.: National Academies Press. http://www.nap.edu/catalog/4962.
Organisation for Economic Co-Operation and Development. (2009). The professional development of teachers. In Creating Effective Teaching and Learning Environments: First Results from TALIS, (pp. 47–86). OECD
Parks, A. (2008). Messy learning: Preservice teachers’ lesson study conversations about mathematics and students. Teaching and Teacher Education: An International Journal of Research and Studies, 24(4): 1200-1216. https://doi.org/10.1016/j.tate.2007.04.003.
Popham, W. J. (2009). Assessment literacy for teachers: faddish or fundamental? Classroom Assessment, 48, 4–11. https://pdfs.semanticscholar.org/0574/3b625f22bad014ccff6facfb1ce7d7164467.pdf.
Popham, W. J. (2011). Assessment literacy overlooked: A teacher educator’s confession. The Teacher Educator, 46(4), 265-273. https://doi.org/10.1080/08878730.2011.605048
Richards, J. C. & Lockhart, C. (1994). Reflective teaching in second language classrooms. London: Cambridge University Press.
Roediger III, H.L., Putnam, A.L., & Smith, M.A.. (2011). Ten benefits of testing and their applications to educational practice. Psychology of Learning and Motivation, 55(1), 1–36. https://doi.org/10.1016/B978-0-12-387691-1.00001-6.
Sapsombat, W., & Roengsumran, A. (2019). A needs assessment study to improve the quality of the Thailand National Test. ASEAN Journal of Education, 5(June), 33–43.
Scarino, A. (2017). Developing assessment literacy of teachers of languages: A conceptual and interpretive challenge. Papers in Language Testing and Assessment, 4(1), 18–40.
Shulman, L. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14.
Sibbald, T. (2009). The relationship between lesson study and self-efficacy. School Science and Mathematics, 109(8), 450–460. http://dx.doi.org/10.1111/j.1949-8594.2009.tb18292.x.
Sitton, P. A. (2006). The effectiveness of lesson study as a professional development model for k-5th grade teachers. Unpublished doctoral dissertation, Capella University.
Verhoef, N., C., Coendersa, F., Pietersa, J. M, van Smaalena, D., & Tall, D., O. (2015). Professional development through lesson study: Teaching the derivative using geogebra. Professional Development in Education, 41(1): 109-126. http://dx.doi.org/10.1080/19415257.2014.886285.
Watanabe, T. (2003). Lesson study: A new model of collaboration. Academic Exchange Quarterly, 7(4): 180-184.
Yalcin Arslan, F. (2019). The role of lesson study in teacher learning and professional development of EFL teachers in Turkey: A case study. TESOL Journal, 10(2), 1–13. https://doi.org/10.1002/tesj.409
Copyright rests with authors. Please cite TESL-EJ appropriately. Editor’s Note: The HTML version contains no page numbers. Please use the PDF version of this article for citations. |