Our project deals with assessment of the Math 811, Arithmetic Review course here at the College of San Mateo. To perform this assessment we are employing several different research techniques. One of these is the focus group. This paper describes in detail our learning about the technique and the four focus groups we conducted in the course of the semester.
At the beginning of Spring semester 2000, we set a goal of conducting two focus groups. We decided on only two because we had never conducted a focus group nor had we read much about how to conduct them. Our research began with reading several articles about the theory and practice of conducting focus groups. We then read the textbook, Focus Groups, a Practical Guide for Applied Research. These sources provided the theoretical basis for how we conducted our focus groups.
Ultimately, we conducted four focus groups during the semester, three for the Math 811 arithmetic course and one for an intermediate algebra course using ALEKS, a computer-based diagnostic and learning program. The ALEKS group gave us further focus group practice and also shed light on the ALEKS program, which was used in a limited way in the arithmetic course as well as in intermediate algebra.
The Procedures and Protocols section largely summarizes what we perceived to be the critical points of Focus Groups, a Practical Guide for Applied Research. In practice, we deviated significantly from some of the guidelines we describe in the Procedures and Protocols section. We deviated largely because we ran fifty-minute focus groups and because, in practice, the single most difficult aspect of conducting our focus groups was getting participants to show up. Limiting our sessions to fifty minutes, one class period, forced us to compact our structure and questions to accommodate the shorter time-frame, but made it easier to get participants to attend. In hindsight, the trade-off was worth it.
The four sections that follow discuss the four focus groups we conducted over the semester.
Definition: A focus group is a one of a series of group meetings in which a homogeneous group of people interact through a series of discussions. The purpose of a focus group is to collect qualitative data from focused discussion.
Having a series of group meetings helps to minimize the effect of inadvertently collecting a group of people that do not provide the desired information or quality information. Such situations can occur when personalities conflict during the discussion; or when the group responses differ significantly from the target population and thus are not as relevant.
Having a homogeneous group is necessary to minimize conflict within the group and to help foster a safe and comfortable atmosphere that will encourage open interaction. Thus, having a series of focus groups individually homogeneous but non-homogeneous across groups is desirable.
The guidelines, key points and scripts provided here are meant to provide some structure from which to begin. It is very important, however, to realize that these are a static and formal set of guidelines that we apply to a dynamic and often very informal setting. The single most dominant source of energy in a focus group lies with the participants. Consequently, the moderator must be prepared to guide but not to control the discussion. The moderator must help to create an environment in which the participants feel free to engage each other in discussion sharing ideas, stories and feelings about the issues that are the focus of the study. Too strict application of the rules stifles the freedom and willingness to share of the group. Rules applied too loosely can lead to interaction that does not provide the information sought. The line that the moderator must walk is fine.
Note: In this case, the goal is a self-regulating group. This is particularly evident in the comments preceded by an asterisk. Here we ask the participants to regulate each other's involvement and to stay focused on the critical issues. With low moderator involvement, participants are not asked about specific issues. Rather, through discussion, participants themselves introduce issues they believe to be salient. Such an approach requires a relatively sophisticated group of participants and is appropriate when the researchers are themselves seeking to more clearly define the critical issues. This approach also requires sufficient time for the participants to develop and define the group dynamics on their own. Probably 1(-2 hours is the minimum time necessary for such an approach.
What follows is a possible script providing instructions to the participants. It is not intended to be read by the moderator but to be paraphrased so as to establish a loose, informal and comfortable environment that will encourage open and free conversation.
We are interested in understanding your thoughts and experiences surrounding your arithmetic/Math 811 course. We are here to learn from your thoughts and experiences. We have several questions to which we would like you to respond. First, a few guidelines we would like you to follow.
Are there any questions?
Poster points - A poster board with a skeleton set of instructions for the participants to refer to during the discussion. The instructions were:
Note: In this script, the self-regulating instructions have been deleted from the previous script. In this case, the group is either not particularly sophisticated and/or there is not enough time to allow the group dynamics to develop on their own a rich and fruitful conversation. This approach is best when the time allotted is less than an hour.
Again, what follows is a possible script. It is not intended to be read by the moderator but to be paraphrased so as to establish a loose, informal and comfortable environment that will encourage open and free conversation.
We are interested in understanding your thoughts and experiences surrounding your arithmetic/Math 811 course. We are here to learn from your thoughts and experiences. We have several questions to which we would like you to respond. First, a few guidelines we would like you to follow.
Are there any questions?
Poster points - A poster board with a skeleton set of instructions for the participants to refer to during the discussion. Same as listed for low-level moderator involvement.
Purpose: To introduce everyone, to identify something common shared by participants, and to get everyone involved in the discussion.
Important: Participants should use only their first names. The moderator should emphasize this to reinforce the desire to keep the participants and their comments anonymous.
Examples: How many units are you taking and how many hours are you working this semester? Where are you working?
Purpose: The introductory question should provide an easy entry into the discussion and foster conversation and interactions among the participants. The question itself is not one that is expected to provide the information desired. However, chosen correctly, the introductory question should generate conversation that could easily and naturally lead the participants to raise the critical issues that are the focus of the study. When this happens, the segue into key issues is a natural extension of issues raised by the participants and not imposed by the moderator.
The introductory question should introduce the general question(s) or issues related to the general question(s), and allow the participants to reflect on past experiences and their connection to current issues at hand. The question should elicit concrete, descriptive responses rather than abstract, theoretical ones.
Examples:
Purpose: These questions should move the conversation closer to the key issues of the study. They should provide a natural transition to the main topics of interest for the researchers. They provide a logical link to more focused issues.
Note: The best way for this transition to take place is for the participants to begin to relate past experiences to present ones on their own. The moderator can then guide the conversation in the desired direction by building off of these comments (see example b). Be sure however, that everyone has had a chance to speak before moving on (see example c).
Examples:
Purpose: These are the questions that drive the study. They are the center-piece around which the focus group is built. The desire to gain insight into these questions motivates the choices of introductory questions, participants and methodology.
Examples:
When your instructor returns a graded assignment or quiz and you have missed some problems, what do you do? Do you go over them with your fellow students? With the instructor in office hours? With a tutor in the Math Resource Center? Do you go over and over the problems you missed, redoing them until you get them correct? If yes to any of these questions, please describe a time when you employed one or more of these methods.
Purpose: These questions should bring closure to the discussion, allow the participants to reflect on what has just transpired, and to clarify previous comments.
Examples:
We offered pizza, drinks and $5.00 gift certificates to all participants. In practice, participants consumed little of the food or drink. However, we believe that this reinforced the importance and appreciation of those who participated.
Date: 3-14-00
Topic: Math 811
Those attending the focus group included Sandra Comerford (moderator), Mike Burke (assistant moderator), Bob Hasson (note taker and technical support, i.e., he turned the tape recorder on and off), and four participants.
The questions were:
Prompt 1 - What value do you see to understanding arithmetic in your everyday life?
Prompt 2 - Are you taking this class to understand mathematics better? Why not?
Prompt 3 - Are you taking this class just to meet college requirements? Why not for understanding?
Prompt 1 - What specifically do you do to successfully complete your homework?
Prompt 2 - What specifically do you do to prepare for quizzes?
Prompt 3 - Do you take part in study groups/partners? Go to office hours? Go to the Math Resource Center? Check and recheck your answers until you get them correct? Check and recheck your answers until you get them correct AND until you understand them? In what way do these help you?
Prompt 1 - What do you do if you missed some problems/questions?
Prompt 2 - Do read and understand the comments your instructor wrote? Does this help? How?
Prompt 3 - Do you go the Math Resource Center or instructor's office hours, or go over the problems with a fellow student? Does this help? How?
Prompt 4 - Do you do the problem over and over again until you can do it correctly and understand it? Does this help? How?
Prompt 1 - Do you understand the format of the course? Explain.
Prompt 2 - Do you understand what happens if you pass or fail a unit exam?
Prompt 1 - Describe an experience when you understood the mathematics.
Prompt 2 - Describe an experience when you enjoyed doing mathematics.
Prompt 1 - Describe an experience when you became frustrated trying to understand the mathematics.
Prompt 2 - Describe an experience when you did not enjoy doing mathematics.
This is a summary of the highlights of the debriefing discussion conducted the day of the focus group. Participants in the post-focus group discussion were Bob Hasson, Mike Burke and Rob Biagini-Komas.
Four students participated in the discussion. One participant (#1) passed the first unit of the course and three (#2-4) did not
Participant #1 studied until she understood the material. Several issues seem critical here. First, she augmented the study habits listed above with others, such as, reading the section and doing problems from that section before it was covered in class. She felt this helped her to understand the material better. Second, her goal was to understand the mathematics, not just to do the homework. She did problems until she felt she understood them. Finally, she seemed capable of determining when she understood the material. This is evidence of a good study skill rather than just a good study habit.
Participants #2-4, on the other hand, seemed to be reciting the party line. The study habits of which they spoke paralleled the habits that their instructor emphasized regularly. The good news is that these students seemed to have accepted these habits as important contributing factors to their success in mathematics. The bad news is that they did not seem to understand that these habits were merely a means to an end, the end being to understand mathematics. This seemed most evident when they were asked why they embraced these habits and they responded ßo they could take the unit exam." They did not seem to have fully appreciated that at a minimum the goal was to pass the unit exam, and that to pass the exam one must understand the mathematics.
Perhaps this suggests a weakness in the program in that the desirable behaviors of doing homework, working together and correcting errors had been effectively conveyed, but that the more critical goal of understanding had not been conveyed. On the other hand, it is probably too much to expect to affect profound changes in a student's academic behavior in a semester let alone a third of a semester. Research suggests that several semesters is a more realistic time frame in which to affect significant change in student behavior.
If one had heard the audio tape of the focus group without knowing who had passed the exam and who had not, it would have been easy to pick out the one who definitely passed. Participant #1 clearly conveyed an understanding of what learning entailed.
This led to a discussion about an instructor from UC San Diego who had created a list of 34 prototypical problems for a calculus course. Students grade in the course depended on the percentage of the 34 problems they could correctly solve/answer.
This ultimately brings one back to defining a course by a set of general goals and guidelines (e.g., desirable student behaviors), specific course objectives (e.g., a list of problems at which students must demonstrate competence), and methods by which these will be achieved and measured (e.g., projects, written work, exams).
Date: 3-23-00
Topic: Math 811
Those attending the focus group included Mike Burke (moderator), Bob Hasson (note taker and technical support), and four participants.
The Questions
Can you describe specific instances by telling a story?
office hours?
Math Resource Center?
working with others?
Can you describe specific instances by telling a story?
go over with students? with instructor? with tutor?
do problems over and over until you get them correct?
Do you know students like this? What drives them to be this kind of student?
Do you know students like this? If (x, y and z) are good things to do, what stands in the way of students doing these things?
This is a summary of the highlights of the debriefing discussion conducted the day of the focus group. Participants in the post-focus group discussion were Bob Hasson, Mike Burke and Rob Biagini-Komas.
Three students participated in the discussion. All three had passed Unit I and all three would pass Unit II the next week.
Question #2-3
One participant did not like the idea of changing instructors. Her new instructor did not use manipulatives like her first instructor and he talked too fast.
This last comment seems to be a red flag. The instructor in question (the one who talks too fast) feels that it is very important that his arithmetic students feel comfortable asking questions, asking him to slow down or asking him to repeat an explanation. He also believed that his students felt comfortable enough with him and the class atmosphere that students will in fact ask when they feel the need. Participant #1's The above comment shows that not all students feel as this instructor hoped and thought they would. Now, the commenting student is from a country where such requests are not made of instructors by students. Thus, one would not really expect her to do so here. However, this instructor completely missed the hidden dynamics of the interactions with this student. So, on a very micro level, the instructor was not as accessible as he thought he was. So informed, this can be, and is being, attended to. Of greater concern is in what other ways is this instructor, or any other for that matter, misassessing his/her abilities, awareness or performance. If we are aware of a problem, we can attend to that problem. But if we are not aware, or have bad information, we can do nothing. Again, this is an area upon which we hope to shed light through this assessment project.
Question #2
One participant talked about different teaching styles between teachers in her home country and those in the United States. Specifically, when adding fractions, students in her country are taught the following rule with no attempt to explain why this rule makes sense:
|
An important question to ask at this point is is it better to teach students the "Find the Least Common Denominator, convert to equivalent fraction and add" method. It is not clear that the answer is yes. The LCD method is a much more complex and difficult algorithm. Many students do not understand what they are doing, nor do they gain any intuition or number sense from such an algorithm. The fact that this process parallels the algebra topic of adding rational expressions is weak justification for adopting the LCD method.
Perhaps, teaching the shorter, easier to memorize rule of the above student and spending all the time saved from learning the multi-step, multi-concept LCD method on developing intuition and numbers sense would be a better use of time. This merits some consideration.
Finally, perhaps the best approach is to take the constructivist approach with all students allowed to develop their own algorithms.
Question #6
Help seeking strategies included getting help in class from the instructor, consulting with other students in class about missed questions on homework or quizzes, redoing problems by oneself outside of class, asking questions in class, going to office hourse for help, and getting help from significant others outside of class.
Question #7 = "The Ideal Student Question"
With respect to ïdeal student" behavior, the participants see students in the math lab, seeking help from the instructor and teaching assistants. With respect to "non-ideal student" behavior, the participants have talked to some students who knew that they did not study enough, that borrowed homework to copy to turn in, and others who work many hours and thus do not have time to study as well as they should. Some of these students recognize what is happening and are attempting to do things better in the future. Finally, the participants see some students who really do not want to be in school, who are not interested in learning, and those who work too many hours to allow them to focus on their studies.
Question #3-4
With respect to the modular format, the participants do not like having to change instructors, although there was also some sense of pluses from changing instructors. The participants like not having to start the whole semester over again if they fail a unit exam. The participants felt that the requirement of passing a unit exam for credit was a valid hurdle.
Date: 4-12-00
Topic: Math 811
Those attending the focus group included Bob Hasson (moderator), Mike Burke (note taker and technical support), and two participants.
This is a critical issue. Part of what we are trying to assess is the value of the modular format. In the big picture, do gains from the modular format out weight the losses. The main strength and purpose of the modular format is to allow students who fail a unit to retake that unit focusing on their specific weaknesses, rather than continuing on to material for which they are not ready. This tends to maintain a higher level of homogeneity within classes comprised of a generally very diverse population. A secondary advantage occurs when the student struggles with an instructor and perceives that as part of the reason that they failed. Changing instructors can provide a fresh start.
On the downside, students tend to want to stay with an instructor with whom they have experienced success. When a student passes a unit and then must change instructors, this tends to undermine their success. While it is good for students to learn that they can be successful with different instructors, students at this level tend to need more successes than additional challenges. In general, the lack of continuity that arises from changing instructors undermines the educational process. Students must get used to new instructors and their routines, and the instructors must get used to new students. Relationships built between student and teacher and broken and must be rebuilt. A teacher that has established a good working relationship with a student may lose the advantages of that connection and knowledge of their strengths and weaknesses.
In general, both students and teachers would prefer not to have students change classes. However, under the modular format, the change is necessary. The question is äre the benefits great enough?"
This leads to the similar question of should an instructor change the composition of groups when using small group instruction. Some research and practice suggests that changing groups regularly is desirable, even necessary. In other instances, the consistency achieved through consistent group composition is perceived as more valuable than the benefits of changing the composition of the groups.
The unsuccessful student tended to be younger, often right out of high school, and not very committed to their education. The unsuccessful student tends not to make use of the resources available. Their home and social environment tend to detract from them achieving academic success. Friends, rather than helping them to study, provide opportunities for them not to study. Home life, rather than supporting the student, may not provide an environment that allows the student to study. The unsuccessful student tends to be less concerned with whether they are doing a problem correctly and more concerned with just doing the homework. They tend not to check their answers, indeed may not have a mechanism by which they can check, and tend not to pursue correct solution very aggressively.
Our goal is to determine for which of these characteristics can we affect change, and how do we do that?
This focus group concerned the use of ALEKS, a web based tutorial program that was used in one section of Intermediate Algebra and also in one module in the Math 811 Arithmetic Review course. We feel that the problems in the use of ALEKS cross course boundaries, so that the Intermediate Algebra usage is relevent to this project.
Date: 5-9-00
Topic: ALEKS and Intermediate Algebra
Those attending the focus group included Rob Komas (moderator), Bob Hasson (note taker and technical support), and four participants.
The participants complained of technical problems. These included problems with their internet service provider, their hardware, and downloading the plug-in required by the ALEKS program. To some degree, all participants had trouble entering their answer into the computer in a manner that ALEKS would accept. These are both very real and serious impediments to implementing ALEKS. These are not sophisticated students. When confronted with these types of problems, they often do not have the problem solving skills, confidence and determination to overcome such obstacles.
The participants made some good suggestions and some not so good suggestions. One of the not so good suggestions was to have online help available for immediate technical support. While the basic concept of online support is good, the emphasis the participant put on ïmmediate technical support" only highlights her lack of problem solving skills. She has neither the patience nor the skills to work through a technical problem.
Two good suggestions included (1) having students work together to provide technical support (not content support since this is individualized instruction) and (2) meeting to do ALEKS as a class several times. Ultimately, in order for the ALEKS to be an effective learning tool, the students must gain confidence in their ability to use the tool and believe that there is value in working with ALEKS. This requires a significant investment in time and energy on the part of the students and instructor. Merely overlaying the program on top of an existing course is not sufficient.
The participants suggested that they have access to a computer-based resource that would allow them to decide what topics they studied, rather than these topics being restricted by their level as decided by the program. This puts the control of what topics a student studies in the hands of the student. Often, students are not capable of effectively making that determination. ALEKS is designed to do this based on an assessment process that the students tend not to be capable of.
Ultimately, the students should be able to get beyond this problem. We as instructors must demonstrate to them the need and power to be able to go from one topic to another without stumbling. And we must demonstrate the interconnectedness of all topics. The overlay of ALEKS onto a course may provide an opportunity to do this.
We are looking for students enrolled in Math 811 to participate in a
group discussion about their experiences with mathematics. The
discussion will last approximately one hour. Lunch and a \$5.00 gift
certificate for Cafe International will be provided for each of the
participants. If you are interested, please complete and return the
form.
1. My name is (please print clearly)
__________________________________________
I am interested in participating in the group discussion AND I am able to meet
¬ Mondays from 11:10-12:00
¬ Wednesdays from 11:10-12:00
¬ Fridays from 11:10-12:00
¬ Tuesdays from 10:10-11:00
¬ Thursdays from 10:10-11:00
2. I am interested in participating in the group discussion but I am
unable to meet during the times listed above. I can meet at the times I
have listed below (give day and time).
______________________________ ______________________________
______________________________ ______________________________
Please check the appropriate boxes.
a. I am ¬ male ¬ female
b. I am ¬ under 20 years old ¬ 20 years or older
c. I have previously had Math 811 under the old format ¬ yes no
d. This is my first semester of Math 811 under the new format ¬ yes no
e. My instructor for the first third of Math 811 this semester was
¬ Church ¬ Komas ¬ Lehmann ¬ Rundberg
March 10, 2000
Dear (participants name),
Thank you for accepting our invitation to participate in the Math 811 discussion group. The discussion will be held Tuesday March 14th from 10:10 AM to 11:00 AM in building 17, room 119. We will provide pizza and drinks. You will also receive a $5.00 gift certificate for Cafe International.
Since we are talking to a limited number of people, the success and quality of our discussion is based on the cooperation of the people who attend. Because you have accepted our invitation, your attendance at the session is anticipated and will aid in making the research project a success.
The discussion you will be participating in will be with other Math 811 students. We are interested in your opinions regarding the effectiveness of the course. All comments made by participants will be completely confidential. If for some reason you find you are not able to attend, please let us know as soon as possible. Mike Burke's telephone number is 650-574-6528 and Bob Hasson's is 650-574-6318.
We look forward to seeing you on March 14th. Thank you.
Sincerely,
Mike Burke and Bob Hasson
May 8, 2000
Dear (participants name),
We are writing you to remind you that the ALEKS discussion group to which you have been invited will meet Tuesday, May 9, in building 17 room 119 from 9:10 AM to 10:00 AM, the hour before class. Pizza and drinks will be served.
We hope you will be present. Your thoughts, impressions and criticisms of ALEKS are very important to us.
Thank you very much.
Sincerely,
Rob Komas and Bob Hasson
May 11, 2000
Dear (participants name),
Thank you very much for participating in the discussion group on ALEKS. Your comments have contributed significantly to our research and to our understanding of the effectiveness of ALEKS and its potential for use in learning mathematics.
If you have any questions please feel free to contact us (Rob Komas at 650-574-6242 or Bob Hasson at 650-574-6318).
Thank you again.
Sincerely,
Rob Komas and Bob Hasson