An appreciation of the role of reflection in professional development is encouraging more and more research scholars, teachers, and teacher trainers to locate research in the classroom. And, most of them collect data using the questionnaire. Given the situation, it is useful to acquire fresh insights into the questionnaire as a research method in L2 situations. Our study demonstrates that respondents may find exploratory questions threatening both cognitively and linguistically, and choose not to respond to them. This will lead to a low response and return rate of the questionnaire. Although a great deal of guidance is available on the design and administration of the research tool, there appears to be very little research on the relationship between the nature of the open-ended questions and the low response rate, especially in the ESL context. The findings of this study show that researchers, investigating classroom-related issues, can use the workshop as an alternative tool for data on unobservable phenomena such as attitudes, perceptions, and opinions. Between the two research methods, the workshop is less threatening than the questionnaire for two reasons. It allows interaction among peers, encourages them to think on the issues through reflective activities, and facilitates their learning from the experience.
So much depends upon respondents in qualitative research. Without their contribution, researchers will not be able to create an authentic knowledge base. For, more often than not, they may be outsiders trying to conduct ethnographic research that includes:
[P]articipant and non-participant observation, focus on natural settings, use of participant constructs to structure the research and investigator avoidance of purposive manipulation of study variables (LeCompte & Goetz, 1982, p. 32).
Given the situation, researchers take their respondents into confidence and try to gain insights into the situation. Acting as facilitators, they create opportunities for the subjects to acquire a meta-awareness of their empirical reality through the use of various research methods. The respondents have a very responsible role to play in this kind of collaborative investigation. While the validity of the study depends upon the findings, these in turn depend upon the respondents’ willingness to reflect upon and ability to analyze the situation and articulate their opinion (in all likelihood, they are unaware of the enormity of their role).
This article examines the efficacy of the questionnaire and the workshop as tools for enabling the respondents to play a responsible role in collaborative and qualitative research.
In India, most ESL research scholars, and students in particular, prefer the questionnaire to any other tool for collecting the data. This appears to be the case elsewhere too (Brown, 2001; Dörnyei, 2003; Nunan, 1992). Indeed, the questionnaire has many advantages; it is inexpensive and easy to administer even to large groups. The data is easy to quantify. The tool creates a sense of security for the novice researchers who are yet to learn their role in collaborative research. More importantly, it will give them an opportunity to know about their subjects before they observe or interview them. Not surprisingly, the questionnaire is mentioned before the interview in the list of research methods.
The questionnaire is used for collecting data on both facts and perceptions. Seliger and Shohamy (1989) say that “in second language acquisition research, questionnaires are used mostly to collect data on phenomena which are not easily observed, such as attitudes, motivation, and self-concepts” (p. 172).
What is important to note in this context is, reflection is a prerequisite for understanding these ‘phenomena,’ and the respondents should be willing to and capable of reflecting on them. First, they have to reflect on action (Schon, 1983). Second, they should be competent enough to articulate their insights in their second language. These two reasons may lead to the “low response rate” which Seliger and Shohamy (1989) identify as one of the main problems that might question the validity of the study. Brown reports that one of the reasons for the low return rate (33.7%) of their questionnaire could be the inclusion of quite a few open questions (2001). Respondents are reluctant to fill in the questionnaire when the questions demand “ quite a bit of writing”, says Dörnyei (2003, p. 48). He thinks that the questionnaire is not an appropriate tool for qualitative and exploratory research, and that it has to be used in combination with other procedures for relevant data. This article describes how we negotiated with the problem (or challenge) when the returned questionnaires had very little data to offer on perceptions crucial to our study.
Our learning from the experience led to the following hypotheses:
Our study examines the nature of hospital administrators’ (HA) workplace communication needs in healthcare setting in the framework of emotional intelligence (Goleman, 1998). We worked with 80 HAs, 20 each from four corporate hospitals--most of them in their mid-twenties. Initially, we decided on three methods for collecting data. The questionnaire was our first choice as it is “self-administered and can be given to large groups of subjects at the same time” (Seliger & Shohamy, 1989, p. 172). Also, it would help raise their awareness of the issues in question. Secondly, it was essential to observe a few HAs at their workplace and record their interactions with the patients and their attendants. Third, we would collect field notes where it was not possible to record the conversations. Although we had considered the interview as well, we thought it might not be possible to interview 80 HAs individually.
We administered the instrument for two very specific purposes. First, we needed demographic data such as age, education, experience etc. Second, we wanted their opinion on their responsibilities as customer relations staff. There were five questions addressing this:
We gave the questionnaire to 40 subjects in two hospitals planning to make any modifications, depending upon the responses, before administering it with the other 40. We were as much pleased with the return rate--100 percent--as we were disappointed by the answers. Very few subjects responded to the five open-ended questions, which were central to the study.
A similar problem was reported in “Surveying Majors in Developing a Capstone Course” (Latiolais, O’Halloran, & Cakebread, 2003). The researchers noted that the open-ended questions in their survey did not yield much useful data:
We also asked them why it was important to become more proficient in these areas. Of the 40 respondents, 27 students answered this question…. They were also asked if they would take such a course.... Only 11 students responded to those questions. Critiques were not significant…. (2003, p. 6)
We had thought that the questionnaire would encourage the subjects to reflect on their experiences in the healthcare setting. What we had not realized was that reflection is a learning process requiring guidance. Schon’s (1987) observations are highly relevant to this context:
[T]he thing I find hardest in the world to do is to teach a student what I know how to do best. For example, to see interesting patterns in data, which I know how to do, I cannot teach my students to do, or I have to work very hard, or I ask myself, "What is it that I’m really doing when I do this?" … I don’t know the answer to it. In order to get the answer I have to actually think about what I do, and observe myself doing it. My theories about it don’t work very well. … [R]eflection-ON- reflection-in action IS an intellectual business, and it DOES require verbalization and symbolization. (p. 5)
The HAs had to “work very hard” to be able to respond to the five open-ended questions. They were not aware of the direction in which to work, having had no training in how to ‘think about what’ they do and ‘observe’ themselves doing it. The HAs, like Schon and the good conversationalists that he mentions, must be “moving between the extremes of observation and activity” while on the job, and without any “verbalization or symbolization.” They need training for this intellectual business without which they will only be able to play a minimal role in collaborative research. In other words, without help, the HAs will not be able to think about and observe themselves let alone find the language to record them in. In light of Schon’s observations, we take a look at three of the questions and present the demands they are likely to have made on the respondent’s willingness to answer them.
1. What skills do you use while managing crises (medical emergencies)?
The respondents must reflect on a few crises, recall the various steps related to the management of different kinds of crises, and then identify the skills that each one of them demanded of them. First, it is not easy to recall how they responded or reacted. What is more difficult for them is to identify the skills involved and find names for them.
2. What is your role in building the image of the hospital?
This, we realized, is a very ambitious question; it demands that the HA-respondents do an insightful review of their work in relation to the mission of the hospital. They have to adopt a holistic approach to the various jobs they do, and evaluate their importance against a set of parameters inherent to the healthcare setting. In addition, they should know the language and an appropriate format to articulate their opinion. In fact, their response may not be very different from what is expected of them in the annual self-appraisal report.
In sum, the HAs may have felt threatened and insecure, as they are ill equipped to handle the question. As teachers who believe in promoting learner autonomy, we might say that if they had reflected on their experiences and evaluated them, they would have had an opportunity to grow as professionals. What we need to remember, many ESL learners prefer to be silent rather than negotiate with the fear of making mistakes. This may be the case with the questionnaire subjects too when they are confronted by open-ended questions which demand opinions.
3. In which situations are you most empathetic/sympathetic?
While the other two questions necessitate a holistic approach, this demands an analytical approach to the HA’s work. The HA-respondents have to reflect (on action) on the various situations in which they interact with patients and their attendants, select a few in which they try to be most empathetic/sympathetic and finally, sum them up briefly in the space provided. More importantly, they have to reflect upon their behaviour and language as both convey empathy/sympathy.
We held several meetings to discuss how to handle this respondent HAs silence. We realized, none of the open-ended questions could be answered without reflecting on and reflecting in action. It is possible the HAs were too busy to spend time on the questionnaire. It is equally possible the HAs need guidance, which the research method does not provide. Instead, it creates a threatening situation making the respondents feel ill equipped, however temporarily, and lonely. It may be relevant to remember, here, the educational experience that people have determines their approach to and perception of their reality. In the same context we must also note that very few educational paradigms break free of the reductionist approach to education in the sense that they continue to promote teacher-dependence subscribing heavily to the banking concept of education (Freire, 1970). This discourages students from adopting a critical and responsible approach to their learning. Very few classrooms and examination boards encourage students to appreciate the link between theory and practice, let alone realize the critical role they have to play in their own learning. When learning is not synonymous with problem solving (Dewey, 1938; Freire, 1978), learners fail to learn how to think and form an opinion, and consequently, schools and colleges spread literacy and not education.
There may be another reason for the respondents’ choosing not to respond to the open-ended questions. Very few people feel comfortable when asked to write down their thoughts. This could be attributed to the lack of opportunities for exploring free writing in the classroom. Most often, the students’ familiarity with various genres remains unutilized. Similarly, there is very little encouragement to experiment with the various social and cultural conventions associated with writing. More importantly, writing in English, in the ESL context, creates a considerably difficult situation. For example, the response to the open-ended question demands a reasonably good command on both the linguistic and textual features of the genre of the formal response. We would like to argue that the questionnaire might demand higher order cognitive and linguistic skills that are usually not considered while drafting the questions. Therefore, the possibility of the questions making the subjects conscious of their inadequacies cannot be ruled out. The inadequacies may include not only conceptual thinness but also a limited command on L2 spelling, punctuation, grammar, and handwriting in addition to the genre of the short answer. In other words, the open-ended questions may remove the subjects from their zone of comfort. They can be unsettling. And, one of the strategies the respondents may use is, to respond to closed questions and leave the open-ended ones.
Indeed, there is a general reluctance to respond to questions that demand writing, as mentioned earlier. People prefer to share their opinions, insights, and questions with others either face-to-face or over the telephone. Spoken interactions bring the interlocutors, however short the exchange is, close. In a dialogue, the researcher can play a supportive role and win the respondent’s confidence. Speech allows the respondent to modify, clarify, and most importantly think through the talk and negotiate for conceptual clarity, with the researcher’s support. Responding to a questionnaire, on the contrary, leaves no room for any such negotiation. Worse still, the answers to the open-ended questions seem to hold the writers to what they have written.
Open-ended questions request the respondent to analyze, critique, evaluate, interpret, and synthesize through critical observation. As researchers, we may think that our study, and our questionnaire in particular, creates the experience essential for the subjects to acquire a meta awareness so they would grow as professionals. But, many subjects seem to find the open-ended questions only threatening. Only a few perceive them as an opportunity to learn and as a result, are likely to respond meaningfully. They fall in the category of the “sage” who “plans, responds to and learns from experience” (Megginson, quoted in Beard & Wilson, 2006, p. 111).
Beard and Wilson made a valid observation regarding the difference between experience and learning,
Educational psychologists define learning as a change in the individual caused by ‘experience’. However, 20 years of experience in a job, for example, does not directly equate to 20 years of learning. How people create and manage their experience is crucial to the process of learning. In order to help people to get the most from experience it is necessary to unleash curiosity so that people actively seek learning, so that they can plan to unveil something that was previously hidden (2006, p. 100).
What the open-ended questions demand is ‘learning’, but what the respondents have is experience. To turn experience into learning, they must learn to reflect.
Given the situation, we had to choose a supportive and non-threatening research method to elicit the responses to the open-ended questions.
According to Seliger and Shohamy (1989):
In using data collection procedures, the researcher has three choices: a) to use ready-made procedures developed and tried out by other researchers; b) to adapt and revise existing procedures; or c) to develop new data collection procedures. (p. 189)
Considering that there were 80 subjects, and the demands on their time, we thought the workshop would be more efficient as a tool than the interview. It would create an informal, non-threatening, supportive forum facilitating reflection through interaction between colleagues (Seibert, 1999).
In an insightful report on “Developing Effective Workplace Learning in UK Hospices: Findings, Issues and Challenges”, Clarke (2004) argues:
[That while] workplace learning is concerned with reflection on and learning from experience, …the recognition of learning also as a social phenomenon extends our understanding of how workplace learning is to be supported. From this socio-cultural perspective, learning is seen as arising as a result of a complex interaction between knowledge acquisition, the development of identity, and practice, based within the work and social activities or practices of groups within organizations. (p. 4)
The workshop creates a forum where through interaction the HAs would explore, modify, and create ideas (Nonaka, 1994) negotiating with the tacit and explicit knowledge bases at their disposal. They could develop a perspective on the issues/questions so as to contribute more meaningfully to the study and the organizational development.
Taking the HR managers of the four hospitals into confidence, we presented how “amongst the healthcare professions, the use of workplace learning is certainly not a new concept” and “informal learning methods that combine reflection on practice have long been seen as key to the development of professional competence” (Clarke, 2004, p. 6).
A brief brainstorming session led to the following differences between the questionnaire and workshop.
Table 1. Questionnaire and Workshop Differences
Led by teacher/researcher
Review one’s own experience
Review the experience in general
Find language on one’s own
Negotiate through interaction
Find insights on one’s own
Negotiate through interaction
After a series of meetings and informal discussions with the individual HR Managers, we managed to schedule the workshops. We were given 90 minutes. We were alerted that we would have to let the HAs leave in case of an emergency. We were also asked to admit any non-HAs that might be interested in the workshop because most of the HR initiatives are open to all the staff. We were pleased to note that a few senior doctors, nurses, and administrative officers came and contributed invaluable insider insights.
The workshop had three aims. These included:
The workshop included individual and group activities. However, most of the time the participant HAs worked in groups. There were 30 in the group: 20 Has, and staff members including the HR manager, doctors, nurses, and other administrative officers. We designed the workshop in such a way that after a few activities, individual participants had to write down the answer to a question. Most importantly, we explained that they could write simple points or phrases in response. We assured them that they did not have to write long answers.
For example, there were five activities to help them reflect on their role in building the image of the hospital. In groups of four, they discussed the logo of the hospital in relation to its mission statement. Then, they, together, designed an alternative logo. They also wrote the mission statement to accompany it. This was followed by their describing their logo and the mission statement to the whole group. Then, we asked individuals to answer the question, “What is your role in building a positive image of the hospital?” They had five minutes, and then we collected their answers. This way, it was possible to obtain reliable data from the HAs. What is important to note here is that the participants had to reflect on this question individually. The activities that preceded the writing activity had only prepared the ground. Following are some of the responses to the question, “What is your role in building a positive image of the hospital?”
a) receiving patients with care and guiding them
b) responding to the problems of the attendants who arrives at reception and helping them in solving their problems
c) providing the patient immediate beds as they arrive and complete the admission at the earliest moment
d) paying more attention and patience to the patients and attendants whatever the situation is.
The reflective activities seem to have encouraged some of the HAs to “symbolize” and “verbalize” (Schon, 1987) their learning. However, it is equally evident from their written responses that most of them preferred the clichéd expressions. Like some of Lee’s subjects, they all “indicated deeper reflections in the oral format than in their written reflections” (2005, p. 712).
Thus, the workshop seemed to have encouraged the participant HAs to contribute a few insights to the study. In this context, it may be relevant to mention Brown’s strategy to improve the return rate:
[Y]ou could go to each of the classrooms and ask the teachers to spend the 15 or 20 minutes necessary for all the students to fill out the questionnaire together. In such a situation, you will have solved problem (1) (which is low return rate italics ours) because the students will be captive audience who will generally feel obliged to fill out the questionnaire. Thus the return rate will be high (2001, p. 7)
In some ways, our workshop may appear to be similar to a group-administered questionnaire. But, in one aspect it was completely different. The participants were not a captive audience, and they participated enthusiastically in every activity--including the writing activity.
More and more research scholars, teachers, and teacher educators have begun to recognize the central role of collaborative research in their professional development. More often than not, they locate research in the classroom (Ali, 2007; Blazquez, 2007; Cosh, 1999; Lee, 2005). Similarly, the encouragement given by various research journals to the teacher’s documentation or reporting of various insights related to the classroom has also led to a healthy interest in classroom practices, especially among non-native teachers of English. The issues may include learner-centred teaching, teacher as a facilitator, supplementary material development, peer feedback, interactive classroom management, testing and washback effect, course review etc. An investigation of any of these issues requires that the teacher take the learner into confidence. Classroom research or collaborative research is meaningful only when both the parties contribute to and benefit from the study.
For instance, if ESL teachers want to adopt a learner-centred approach to teaching, they may try to collect data on the students’ willingness and competence to accept responsibility for their own learning. But, if the students do not know the various features inherent to their role in a learner-centred classroom, they may not be able to say anything significant. Similarly, if teachers want to collect data on how students rate themselves as listeners/speakers/readers or writers, they may not find the questionnaire a useful tool to collect data. The learners must know what each of these skills involves and relate the knowledge to their own practices to be able to rate themselves. Preferences about peer and teacher feedback is another topic on which teachers may try to collect data using the questionnaire. In a culture that encourages teacher-dependence, students may not appreciate peer feedback, especially when they are not aware of its advantages. And, hence, the questionnaire may preempt the possibility of the learner growing through the experience of reflecting on peer feedback. Whereas a workshop that hinges upon activities built around peer feedback may give the learner-subject an opportunity to reflect on it and change perceptions accordingly. In other words, there may be situations where the student-subjects have to think critically about their experiences and present their perceptions. It is in such contexts that a questionnaire may not give the necessary guidance and yield no useful data. Therefore, teachers need to keep the various features of their L2 situation in mind, and arrive at a list of “do’s and don’ts” relevant to their specific contexts.
In the ESL situation, the language might also create barriers to reflection, and hence perception. As the written responses to our questions show, even adult ESL learners need a great deal of encouragement before they are willing to write. They may prefer to give their reflections orally. Therefore, the teacher-researcher has to be certain about the nature of the research method and the level of its difficulty in not only conceptual but also linguistic terms for the learner. However attractive the format may be, if it leads to insecurity or a feeling of inadequacy, the method will fail to generate any meaningful data. Hence, the teacher-researcher has to be equally certain whether the data has to include the description of an experience and/or learning. Many students manage to describe their experience. But, if they are asked to write about their learning, they may not have the language for doing it. Then, the teacher may find the workshop more relevant than a questionnaire as it will prepare the respondents to participate responsibly in the investigation. This would mean identifying the directions in which the students have to reflect and then, design a few activities to guide them through their reflection.
One of the points our workshop highlighted was the necessity of raising the participants’ consciousness so as to help them acquire a perspective on the experience. When different students examine the experience, there is a likelihood of the emergence of a wider perspective. The participants in a workshop will be able to examine the experience in its totality, and relate it to their own role in it. This understanding of their role is to be considered as their learning. This enables the student-subjects to collaborate with the teachers in research.
The teacher is the best judge when it comes to assessing the students’ conceptual and linguistic abilities. These should be the criterion for the choice of the research method that supports reflection. Given the situation, in addition to the reasons already discussed, what may make the workshop a good choice is the fact that it can be conducted with the whole class during class hours. Equally important, the teacher can identify the problems students may have in trying to learn from the experience. We could take such factors into account while involving our students in collaborative research. And, if the workshop is chosen as the research method, we can be more sensitive to these factors and address them during the process of reflection itself.
Our study gives a fresh perspective on what is fundamental to research--research methods. It illustrates that we need to be critically aware of these instruments through out our research as, evidently:
[O]ne potential problem with deciding too far in advance is that (y)our choice may limit (y)our ways of thinking about and studying a particular problem or issue. Indeed, the strongest researchers may be those who remain flexible…be able to adapt their research methods to the issues they are trying to learn about, and to the places their research leads them. (Brown & Rodgers, 2002, p. 18)
Premakumari Dheram teaches in the English and Foreign Languages University (formerly CIEFL), Hyderabad, India. Her research stems from a range of intriguing issues in collaborative classroom dynamics, and her work is published in several ESL journals. She is the co-editor of this special issue.
Nitya Rani is an English language professional with experience in English language teaching and soft skills training. She has also been engaged in e-learning and content development; edited distance teaching materials and supervised project reports, has combined experience of writing, freelancing as a print journalist, and producing documentaries on video and script-writing for educational TV. She has also been involved with project management of teams comprising contributors, including SMEs, language experts, editors, reviewers, designers, and developers, in addition to her doctoral research. Her current engagements include repository building for the developmental sector.
Ali, S. (2007). Reflective teacher observation model for in-service teacher trainees. English Teaching Forum 45(1), 16-25
Beard, C. & Wilson, J.P. (2006). Experiential learning. London: Kogan Page, Ltd.
Blazquez B. A. (2007). Reflection as a necessary condition for action research. English Teaching Forum 45 (1), 26-34.
Brown, J.D. (2001). Using surveys in language programs. Cambridge: Cambridge University Press.
Brown, J.D. & Rodgers, T.S. (2002). Doing second language research. Oxford: Oxford University Press.
Clarke, N. (2004). Developing effective workplace learning in U.K. hospices: Findings, issues and challenges. Retrieved 19 February 2007 from: [http://www.ugbs.org/weru/pub/WorkplaceLearningReport.pdf]
Cosh, J. (1999). Peer observation: A reflective model’. ELT Journal 53 (1), 22-27
Dewey, J. (1938). Experience and Education. (New York. Collier Books).
Dörnyei, Z. (2003). Questionnaires in second language research. Mahwah, New Jersey: Lawrence Erlbaum.
Freire, P. (1970). Pedagogy of the oppressed. (M.B. Ramos, Trans). New York: Penguin Education (originally published in 1968).
Freire, P. (1978). Pedagogy in process: The letters to Guinea-Bissau. New York: A Continuum Book, Seabury Press.
Goleman, D. (1998). Working with emotional intelligence. New York: Doubleday/Bantam.
King, P.M. & Kitchener, K.S. (1994). Developing reflective judgment: Understanding and promoting intellectual growth and critical thinking in adolescents and adults. San Francisco: Jossey-Bass Publishers.
Latiolais, M.P., O’Halloran, J. & Cakebread, A. (2003). Surveying majors in developing a capstone course 1-7. Retrieved 10 February 2007 from: [http://www.maa.org/SAUM/new_cases/new_case_11_03/assessPSU2r2.html].
LeCompte, M.D. & Goetz, J.P. (1982). Problems of reliability and validity in ethnographic research. Review of Educational Research 52(1), 31-60.
Lee, H-J. (2005). Understanding and assessing preservice teachers’ reflective thinking. Teaching and Teacher Education 21, 699-715.
Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science 5 (1), 14-37.
Nunan, D. (1992). Research methods in language learning. Cambridge: Cambridge University Press.
Schon, D. (1983). The reflective practitioner. New York. Basic Books.
Schon, D. (1987). Educating the reflective practitioner. Presentation at the meeting of the American Educational Research Association Washington, DC, April 13. Retrieved 2 February 2007 from: [http://educ.queensu.ca/~ar/schon87.htm].
Seibert, K.W. (1999). Reflection in action: Tools for cultivating on-the-job learning conditions. Organisational Dynamics (Winter), 4-65.
Seliger, H.W. & Shohamy, E. (1989). Second language research methods. Oxford: Oxford University Press.
© Copyright rests with authors. Please cite TESL-EJ appropriately.
Editor's Note: The HTML version contains no page numbers. Please use the PDF version of this article for citations.