The Decision Sciences Journal of Innovative Education

Introduction

Doctoral level quantitative courses present a challenge for both students and academics, due to the typically wide variety of student skills and their varying interests. This paper discusses the redesign of one such course to teach the research skills required by students, and at the same time, to introduce students to a wider than usual range of statistical techniques. By using a problem based approach, the course combines material usually taught in research methods classes, such as research question specification and data collection, and also material which is usually restricted to advanced statistics classes, such as complex multivariate techniques. Feedback suggests that this approach is seen by students as more useful than a traditional techniques based course.

The challenge

Many students dislike statistics/quantitative methods courses, and see such courses as an obligatory hurdle, requiring them to pass a course in which they have limited interest or ability. Resistance to statistics/quantitative methods courses stems in part from the fact that most such courses teach proficiency in a limited number of statistical techniques, outside the context in which they are used. Even research students, who might be expected to be more interested in quantitative courses, usually take them early in their studies, when students are unsure or unaware of what statistical methods (if any) they will use. The statistics course can then become a tedious exercise in mastering techniques which are rapidly forgotten after the conclusion of the course.

Three years ago I was asked to take over the 40-hour doctoral level quantitative methods course at my business school. Most of the students have limited experience in statistics, and many are self-confessed ‘math phobics’. The course had previously been run as a conventional techniques based course, but had been criticised by students and supervisors as not providing adequate preparation for doctoral work. So I started by surveying supervisors and current and past students about what they wanted in the course, what they liked, and what they did not like, about quantitative methods courses. The results reinforced that students prefer to learn how to identify techniques to solve their research problems, as opposed to learning a few techniques in detail. They expect to seek expert statistical advice, if they need it, for their analyses. Worse, the ‘math phobic’ students didn’t want to do a quantitative course at all, so generating interest in the course presented a challenge.

Developing a problem based course

I decided to redesign the course, basing it on how research students use statistical techniques, as solutions to research problems, rather than providing depth in a limited number of techniques that they might or might not use. The course aims to teach, and provide practice in, the skills that students will need for their theses: the ability to design data collection, select, perform, interpret and report statistical analyses. At the same time the range of techniques that are covered in the course could be increased, by decreasing the depth in which some techniques were covered.

The course begins with a discussion of the doctoral requirement to make a ‘contribution’, and we discuss what is a ‘sufficient’ contribution. Most, though not all, students have some idea about their ‘research question’ and we discuss how they can try to determine if their research will make a ‘sufficient’ contribution. Students develop specific, testable research questions, which leads on to a discussion of the variables of interest, and the measures that can be used to quantify these variables. (Experience shows this is very difficult, but also useful, for early stage doctoral students, who typically have broad and untestable research questions.) This first part of the course aims to start students thinking about what they need to do to test their research question, an issue which is critical, even for students who plan an entirely qualitative thesis. Students love this section, because it relates directly to their research. Importantly, it also creates interest in the next stage, the data and tests they need to investigate their research questions. After the first session students are asked to fill in an optional quiz, which presents them with problems in selection and interpretation of common statistical techniques. The quiz allows me to assess the varying levels of ability in the class, so I can stretch the best students and help the weakest.

My academic colleagues had identified that many students who had completed traditional quantitative courses had little skill in data collection, particularly in developing surveys. As a result, the course covers questionnaire construction in detail, with the aim of ensuring that students do not spend substantial amounts of time and money on a survey that receives a poor response rate, or worse, where they achieve a high response rate, but find that they have failed to include critical variables. Students often naively believe that they have the skills for data collection, though they expect to seek help with statistical analysis. This section of the course reinforces that it is absolutely crucial that data collection is done well, because no one wants to re-do it later. Critical issues such as entering and cleaning data are addressed, because despite their importance, few students have any idea how to do this.

Having covered research question specification, variable selection, and data collection, we discuss hypothesis formulation and testing as methods of investigating the research question. Most students have covered these areas before, but when tested, have little ability to formulate a specific hypothesis, let alone interpret a p value. The course then moves on to techniques with a problem based approach. We start with a series of research problems: “I want to compare the performance of these two or three groups; to assess whether there is a relationship between a and b; to assess whether a predicts b…”. We discuss the statistical tests that can be used to address these problems and the type of data necessary to perform the tests. For example a comparison of two groups might involve one of two sorts of t tests, a chi square test, a regression or structural equation model, depending on the data that has been gathered. This reinforces the importance of an understanding of analytical skills at the data collection stage. Even students who intend to do only qualitative analysis can become very interested in the material as they realise that they need to become ‘intelligent consumers’ of quantitative material in the literature.

The course covers simple statistical tests (e.g. t-tests, ANOVA, regression), in detail. Questions from the earlier quiz are used as examples throughout the course, and students appear to delight in recognising that they now understand the correct answers, providing encouraging feedback on their progress. Progressive small assignments give students practice in selecting an appropriate test, performing the test, recording, interpreting and writing up the results, all placed in the context of a solution to a statistical problem. Even the ‘math phobic’ students appear to derive substantial satisfaction at their increasing ability to select and perform tests as solutions to problems. This then lessens their resistance to further, more complex, techniques.

The course then rapidly moves on to cover a wide range of multivariate techniques that are usually restricted to advanced courses, such as CANCOR and structural equation modelling. Students are not expected to be able to perform these techniques, as this is beyond the scope of the course, and would be irrelevant for most students. The emphasis, in contrast, is on recognising applications for which these tests are useful. This superficial coverage of a wide range of techniques is very unusual, but a critical component of the course. Knowing what techniques are available allows a student to specifically investigate a technique if and when it is necessary, rather than spending a large amount of time learning complex techniques in detail. Whether academics like it or not, the vast majority of doctoral students seek help with their statistical analyses, from peers, supervisors or statisticians. A student who knows that a technique such as CANCOR exists will be in a much better position to ask for, and obtain, appropriate advice than a student who only knows that they have a large number of variables that are associated in a complex fashion.

As well as the progressive assignments to develop mastery of simple techniques, students complete two major assignments. The first helps students to practise the skills involved in a major thesis, requiring them to specify a research question and hypotheses, collect data, choose and perform appropriate analysis, record and interpret the results, and write-up the analysis as a research paper. This exercise integrates the different topics of the course, and gives the students practical experience in the skills they will require for their theses and to publish. The second major assignment offers students a choice of topics, depending on their interest. The first option (typically chosen by students who intend to pursue a quantitative thesis) requires them to investigate the use of an advanced multivariate technique in detail, giving them increased depth in a technique that they expect to use. The second option, usually chosen by students intending to pursue a qualitative thesis, requires them to review the use of a technique in a published article in their field, encouraging them to be competent readers of quantitative material. The assignments require substantial work, but a post-course survey of students has shown strong support for the assessment, with students recognising that they learn substantial amounts from the different types of assessment.

The course is still evolving, but feedback has been very good. Partly on the basis of the development of this course, the author was the recipient of a University Award for Teaching Excellence. Privacy requirements mean that student evaluations cannot be directly compared with the former course, taught by another academic, but the course has gone from the least popular doctoral course to one of the most popular, with a student evaluation of 4.53 on a 5 point scale on the anonymous post-course evaluation. Sample qualitative student feedback has included: “Just fantastic – a great module” and “Excellent course with very valuable walkaway knowledge”.

Conclusion

In summary, the course reflects a marked change in teaching quantitative methods to doctoral students. The course uses a problem based approach to integrate components of research methods and quantitative analysis that are important to doctoral students. By placing statistical analysis in the broader context of addressing a research question, collecting and analysing data, recording, interpreting and writing up results, the course has substantially increased student learning and made the material relevant and useful for the students’ theses.