Assembling the data on data-driven learning: A meta-analysis of design issues and outcomes
Résumé
Corpus linguistics is, by its very nature, an applied field, with potential relevance in any area which deals with text in one form or another. One of the earliest such applications was in language teaching, where it is commonly associated with the work of Tim Johns (1990). He coined the phrase data-driven learning (DDL), which is still commonly used as a cover term for any use of language corpora or associated corpus linguistic tools or techniques for L2 users. The last 20-odd years have seen considerable output of academic papers in the area, outlining possible activities, describing actual courses, explaining and justifying the rationale, and so on. The obvious question, of course, is: "Does it work?" There are several ways of addressing this, the most obvious being to conduct an original study to collect new data on specific questions. As individual studies accumulate, however, some kind of overview becomes necessary to gain a broader picture of the field as a whole. Considerable work in applied linguistics over recent years has sought to find ways to make such research synthesis as systematic as possible, and can generally be divided into two broad streams: narrative synthesis and meta-analysis, each with its own advantages and disadvantages (see the papers in Norris and Ortega 2006 for an overview). The narrative synthesis is not unlike the traditional 'literature review' which features in the introductions to many academic papers. It differs in avoiding the narrow focus of a specific topic, and attempts a systematic trawl of all relevant publications, thus reducing the subjective selection of papers for consideration. But it potentially falls down on the rigour of the analysis itself, which can retain some of the problems inherent in literature reviews (e.g. Boulton 2010 on learning outcomes of DDL; Boulton 2012 on DDL in English for Specific Purposes). The meta-analysis involves essentially the same systematic collection of papers, but focuses exclusively on quantitative studies, thus neglecting the value of qualitative studies (for an overview, see Richards 2009). It is thus less broad in coverage than the narrative synthesis, and the analysis is potentially reductive and simplistic, lumping together of all types of specificities of individual studies. Its advantage is that it allows a pooling of the quantitative data from all relevant studies available. This paper presents a meta-analysis of DDL studies (cf. the preliminary work in Cobb and Boulton forthcoming). The work is still in progress, but to date we have collected 140 papers which seek to evaluate some aspect of L2 corpus use, of which 21 provide suitable quantitative data - minimally, means and standard deviations deriving from pre/post-tests and/or experimental/control groups. Work so far suggests a substantial effect size, currently standing at 1.42 standard deviations. Focus on a single effect size figure can be strategically or politically expedient (cf. Grgurović et al. 2013), but meta-analysts are keen to go beyond this to avoid a reductionist picture in such a complex area as language learning (cf. Larsen-Freeman and Cameron 2008). This paper thus seeks to situate the study, with the focus correspondingly not only on the outcome itself but also on the issues raised in collecting and selecting the studies for inclusion, as well as in analysing and sorting the resulting data. Due consideration is given in particular to the deliberately broad definition of DDL, which we have taken to include all uses of corpora by non-native speakers. This seems to be compatible with Johns' original vision, and since DDL clearly means a range of different things to different people, it seems sensible to begin with a broad sweep. We also discuss the inclusion / exclusion criteria in the selection process: we make no distinction between papers appearing in prestigious peer-reviewed journals and elsewhere - smaller journals, book chapters, conference proceedings, as well as 'grey' literature in the form of unpublished doctoral theses (though we exclude research which has not been formally written up, such as unpublished conference presentations or slides). This should ensure that quality work published outside mainstream sources is not ignored, and that negative outcomes in particular are less likely to be overlooked (Oswald and Plonsky 2010). Some meta-analyses have introduced weighting systems, though we have initially attempted to avoid such a priori judgements. Other issues arise from the pooling of quantitative data from highly varied studies - in other words, are the studies sufficiently similar that their results can be legitimately pooled at all? Though we argue that we are not comparing apples and oranges in the overall meta-analysis, the studies can usefully be grouped into different sub-categories for more in-depth understanding, and in one of two ways. In terms of research design, particular importance is accorded to the distinction between the effectiveness of a treatment (as measured by within-groups pre/post-tests: ES = 1.68; d = .84) and its relative efficiency (between groups: ES = 1.04; d = .73). In terms of research questions, it is also possible to derive a number of sub-categorisations allowing meta-analyses of subsets depending on more focused topics, thus allowing greater depth of understanding on more specific issues. As in corpus linguistics, raw data and statistics are useful, but they need interpretation and contextualisation to become meaningful. A careful meta-analysis, with transparent inclusion criteria and sensitivity to individual differences between studies, provides one way of combining both, underlining the importance of effect size in relation to statistical significance (cf. Duff et al. 2007). As such, it allows us to go some way towards overcoming the fragmentation of the field and to provide some kind of evaluation of the state of research in DDL as a whole. In devising more focused subsets of studies, we are able to adopt a realistic evaluation (Pawson and Tilley 1997) and address not just the question of 'Does it work?', but how effective and efficient it might be in different forms for different learners for different purposes in different circumstances. References Boulton, A. 2010. "Learning outcomes from corpus consultation." In M. Moreno Jaén, F. Serrano Valverde and M. Calzada Pérez (eds.) Exploring new paths in language pedagogy: lexis and corpus-based language teaching. London: Equinox, p. 129-144. Boulton, A. 2012. "Corpus consultation for ESP: a review of empirical research." In A. Boulton, S. Carter-Thomas and E. Rowley-Jolivet (eds.) Corpus-informed research and learning in ESP: issues and applications. Amsterdam: John Benjamins, p. 261-291. Cobb, T. and Boulton, A. 2014. "Classroom applications of corpus analysis." In D. Biber and R. Reppen (eds.) The Cambridge handbook of corpus linguistics. Cambridge: Cambridge University Press. Duff, P.A., Norris, J.M. and Ortega, L. 2007. "The future of research synthesis in applied linguistics: beyond art or science." TESOL Quarterly 41: 805-815. Grgurović, M., Chapelle, C.A. and Shelley, M.C. 2013. "A meta-analysis of effectiveness studies on computer technology supported language learning." ReCALL 25 (2): 165-198. Johns, T. 1990. "From printout to handout: grammar and vocabulary teaching in the context of data-driven learning." CALL Austria 10: 14-34. Larsen-Freeman, D. and Cameron, L. 2008. Complex systems and applied linguistics. Oxford: Oxford University Press. Norris, J.M. and Ortega, L. (eds.) 2006. Synthesizing research on language learning and teaching. Amsterdam: John Benjamins. Oswald, F.L. and Plonsky, L. 2010. "Meta-analysis in second language research: choices and challenges." Annual Review of Applied Linguistics 30: 85-110. Pawson, R. and Tilley, N. 1997. Realistic evaluation. London: Sage. Richards, K. 2009. "Trends in qualitative research in language teaching since 2000." Language Teaching 42 (2): 147-180.