Digital Methods
Digital Assessment
Digital methods used to investigate “public contestation over topical affairs” (Marres, 2015, p. 9) – or in this case, the absence of debate.
“What’s intelligence? That’s what my tests measure!” (apocryphal remark attributed to Alfred Binet, inventor of the IQ test
‘sites where literal and not simply metaphorical calculations are made possible by the mathematical or at least arithmetic format of the documents being brought back and forth’ (Latour, 2005, p. 181)
Actor-network theory and praxiography constitute the principal theoretical resources while trace ethnography and software studies were the methodological approaches used to follow the distributed agency.
“Combines the richness of participant-observation with the wealth of data in logs so as to reconstruct patterns and practices of users in distributed sociotechnical systems.” (Giger and Ribes, 2011)
“Code, woven into the background of transactions, habits and perceptions, does not often become visible, except in breakdowns, failures and at certain other moments.” (Mackenzie, 2006, p. 170).
1: How are the technologies that e-assess adult literacy skills enacted in European educational policy and practice?
2: How are standards made and deployed through local and international e-assessment networks?
3: How are data about people produced through e-assessment events?
The so-called ‘data revolution’ has been transforming many aspects of our lives. Ranging from the ‘microdata’ used to construct portraits of individuals to the aggregated models of big data, the technologies of data production have been assuming an increasingly important role in educational practices. These have been accompanied by new ways of visualizing, sharing and presenting digital data that have in turn have been influencing how educational attainment is represented, managed and, performed.
Much of this is due to novel digital technologies that, in addition to intensifying older data production techniques, allow for data to be produced and distributed in larger quantities and more quickly than ever before. The pervasive and voracious character of digital data production means that it acts as a constant gardener patiently ‘harvesting' data on the ability, behaviour, identity and habits of people through measurement and classification.
As educational practices are becoming increasingly digital, new ways of tracing and understanding the contingencies, hiatuses, compromises, and controversies that come with the folding of social and technical actants into the everyday courses of action that constitute education are necessary.
It describes a methodological approach, trace ethnography, that allows researchers to follow the distributed agency of digital actants such as code.
Sites where digital assessment practices are performed are explored as fruitful areas of investigation. These oligopticons provide narrow but extremely detailed views of individuals and collectives that are slowly built up log-file by log-file, data point by data point.
The presentation concludes with suggestions for how researchers, practitioners and other stakeholders can make decisions informed by an understanding of how data are produced by the various epistemic communities that perform the work of data production.
"A Simple Equation: More Education = More Income"
"Americans Think We Have the World’s Best Colleges. We Don’t."
"It’s Now the Canadian Dream"
"The United States, Falling Behind"
"U.S. Adults Fare Poorly in a Study of Skills"
"Stubborn Skills Gap in America’s Work Force"
Literacy
Competencies and/or Skills
Human Capital
Mulilingual Literacy
Language Skills or Return on Investment (ROI)
The post-war project witnessed a convergence of several phenomena: of comparative education. Psychometrics used as a way to create a measurable construct: literacy. This in turn was linked to statistical analyses that were able to produce data for macro-econmics that were able to be translated into skills and human capital.
Assessment as a tool
Assessment as data /artefact producer
However, assessment can be understood and investigated as a practice
(yes)
"An unpredictable and unstable processes of ‘translation’ as various actors and objects come together to form a network that constitutes particular settings and events" (Maddox, 2014, p. 2)
Praxiography - "starts with practices, situations, and events in which information technologies appear, asking openly what occurs and what emerges" (Strand, 2010, p. 5)
One 'local smal- scale assessment site: Organization Z.
One international large-scale international assessment site: The Organisation for Economic Cooperation and Development (OECD)
"Code is a set of permutable distributions of agency between people, machines and contemporary symbolic environments carried as code. Code itself is structured as a distribution of agency" (Mackenzie, 2006, p. 19).
Traces are the digital artefacts that indicate the existence or passing of something or someone. This can refer to:
PIAAC
The Programme for the International Assessment of Adult Competencies (PIAAC) is run by the OECD and attempts to render quantifiable and commensurable the range of literacy practices of over ‘166,000 people representing more than 724,000,000 adults in 24 countries/economies between the ages of 16-65.
A network graph depicting the PIAAC network
I am going to ask some questions about ^CurrentLast
If (B_D12h=1 and B_Q12a=1) then ^CurrentLast=’your participation in open or distance
education.’ else if (B_D12h=1 and B_Q12c=1) then ^CurrentLast=’your participation in organised sessions for on-the-job training or training by supervisors or co-workers.’ else if (B_D12h=1 and B_Q12e=1) then ^CurrentLast=’your participation in seminars or workshops.’ else if (B_D12h=1 and B_Q12g=1) then ^CurrentLast=’your participation in courses or private lessons.’ else if (B_D12h=2 or B_D12h=4) then ^CurrentLast=’the activities you just reported on. I will start with some questions on the last of these activities that you participated in.’
INTERVIEWER: Press <Next key> to continue
ROUTING: If (B_D12h = 1 and B_Q12c = 1), go to B_Q14b else if (B_D12h = 1 and (B_Q12c = 2 or
B_Q12c = DK or B_Q12c = RF)), go to B_Q14a else go to B_Q13
OECD, 2010, p. 27)
ln yin =μn + β1Cin+ β2 (CinNn ) + β3Ein + β4 E2in+ β5 Gin + εin
ln yin = the hourly wage of the individual (i)
μn = country fixed effects (n=1,..., 22 countries)
β1 = earnings gradient associated with measured human capital
Cin = individual skill measure
β2 = coefficient of interest
Nn= various measures of country characteristics
β3Ein = years of actual labour-market experience
β4 E2in
β5 Gin= gender indicator
εin = stochastic error
Organisation Z
Organisation Z (a pseudonym) is a network of language schools in France, Spain, and Germany. The head office, situated in Paris, has for its primary activity the provision of Vocational Education and Training (VET) to adults (18-65 years). It positions itself as a node in the distribution and organisation of learning software and teachers of English to Speakers of Other Languages (ESOL).
Same network with more actants
Map of depicting interactions between teachers, a learners, and Shareable Content Objects (2012).
function sendScore(score, min, max) {
// NB : toString is necessary to keep float values from being rounded.
doLMSSetValue(“cmi.core.score.raw”, score.toString());
doLMSSetValue(“cmi.core.score.min”, min.toString());
doLMSSetValue(“cmi.core.score.max”, max.toString());
doLMSSetValue(“cmi.core.lesson_status”,”completed”);
***
€80 000 000 of Personal Training Accounts
Education & Skills Online has been developed as an assessment relevant to youth and adults of all ages. Institutions, organisations or local governments can use the online tool to assess the skills of a particular population with the goal of providing training or for research purposes.
Assessment stories 1 & 2
Brink, Roelien, & Lautenbach, Geoffrey. (2011). Electronic assessment in higher education. Educational Studies, 37(5), 503-512. doi: 10.1080/03055698.2010.539733
Central Statistics Office Ireland. (2012). Blaise and PIAAC: Building a survey management system for PIAAC with Blaise. Paper presented at the IBUC 2012 14th International Blaise Users Conference, Strand Palace Hotel, London. http://www.blaiseusers.org/2012/ppt/03b.pdf
Desrosières, Alain. (2006). From Cournot to Public Policy Evaluation: Paradoxes and Controversies involving Quantification. Prisme, 7.
Geiger, Stuart, & Ribes, David. (2011). Trace ethnography: Following coordination through documentary practices. Paper presented at the 44th Hawaii International Conference on System Sciences (HICSS), Hawaii.
Hanushek, Eric, Schwerdt, Guido, Wiederhold, Simon, & Woessmann, Ludger. (2013). Return to Skills around the World: Evidence from PIAAC. Paris: OECD.
Harman, Graham. (2009). Prince of Networks: Bruno Latour and Metaphysics. Melbourne: re.press.
Latour, Bruno. (2014). An Inquiry into Modes of Existence. An Anthropology of the Moderns. Glossary. from http://www.modesofexistence.org/inquiry/#a=START+UP&s=0
Østerlund, Carsten, Sawyer, Steve, Ribes, David, Shankar, Kalpana, & Geiger, Stuart. (2014, 4-7 March). What to Do with all those Traces People Leave behind: Computing, Culture, and (Bits of) Context? Paper presented at the iConference 2014, Berlin.
Mackenzie, Adrian. (2006). Cutting Code. Software and Sociality. New York: Peter Lang.
Maddox, Bryan. (2014). Globalising assessment: an ethnography of literacy assessment, camels and fast food in the Mongolian Gobi. Comparative Education, 1-16. doi: 10.1080/03050068.2013.871440
Mol, Anne Marie. (1999). Ontological politics. In J. Law & J. Hussard (Eds.), Actor Network Theory and After (1st ed.). Oxford: Blackwell Publishers.
OECD. (2010). PIAAC Background questionnaire MS version 2.1 d.d. 15-12-2010. Paris: OECD.
OECD. (2013). Skills Outlook Key Findings. Paris: OECD.
Ribes, David, Jackson, Steven, Geiger, Stuart, Burton, Matthew, & Finholt, Thomas. (2013). Artifacts that organize: Delegation in the distributed organization. Information and Organization, 23(1), 1-14. doi: 10.1016/j.infoandorg.2012.08.001
Simondon, Gilbert. (1980). On the Mode of Existence of Technical Objects (N. Mellamphy, Trans.): University of Western Ontario.
Stödberg, Ulf. (2011). A research review of e‐assessment. Assessment & Evaluation in Higher Education, 1-14. doi: 10.1080/02602938.2011.557496
Venturini, T., Baya Laffite, N., Cointet, J. P., Gray, I., Zabban, V., & De Pryck, K. (2014). Three maps and three misunderstandings: A digital mapping of climate diplomacy. Big Data & Society, 1(2). doi: 10.1177/2053951714543804
Williamson, Ben. (2014). New Centers of Data Visualization in Education. Retrieved June 26, from http://dmlcentral.net/blog/ben-williamson/new-centers-data-visualization-education
Zabal, Anouk , Martin, Silke, Massing, Natascha, Ackermann, Daniela, Helmschrott, Susanne, Barkow, Ingo, & Rammstedt, Beatrice. (2014). PIAAC Germany 2012: Technical Report. Münster/New York: The Federal Ministry of Education and Research.
