Measuring beyond numbers: Navigating the complexities of digital skills measures

Measuring beyond numbers: Navigating the complexities of digital skills measures


By Anna Cristina D’Addio and Divya Sharma, GEM Report, and Frank Reichert, Hong Kong University 

In a world without measurement, everyday tasks would lack precision and reliability. From baking a cake with the right amount of ingredients to managing finances, measurement helps ensure accuracy, maintain consistency, and make informed decisions. We now attempt to quantify happiness, gauge the state of political rights and civil liberties, and assess the overall quality of life and education through indicators. But there’s a catch: while these measurements provide invaluable insights, they often overlook context 

Consider the example of digital skills measures, particularly in the context of Sustainable Development Goal global indicator 4.4.1. These measures most typically revolve around activities associated with digital skills, e.g., adding attachments to an email, or copying and pasting a file. These are approximate measures. The 2023 GEM Report on Technology in Education highlights some of these challenges: comparability over time, validity of comparisons between students by gender and socio-economic status, and difficulty keeping up with technological advancements. 

Recognizing digital skills’ multidimensional nature, the International Telecommunication Union (ITU) and the UNESCO Institute for Statistics (UIS), co-custodians of SDG global indicator 4.4.1, have emphasized the need for a robust framework. The ITU Expert Group on ICT Household Indicators has adopted the five competence areas from the Digital Competence Framework for Citizens (DigComp) as the basis for this indicator. The resulting composite indicator is an adaptation of EU’s Digital Skills Indicator (DSI) and is now officially an SDG 4.4.1 indicator. From 2023, extra digital activities have been included to balance the assessment of digital skills better. Measuring new things that did not exist before and about which we did not need to know anything before adds complications.  This means that what should be measured and how needs to be updated continuously. This blog contains new initiatives to do so since the 2023 GEM Report went to print.

At the national level, in France, the Directorate of Evaluation, Foresight and Performance (Direction de l’Évaluation, de la Prospective et de la Performance, or DEPP) conducted its inaugural evaluation of students’ digital skills after secondary school based on 16 competences of the DigComp framework, distinguishing between different performance levels. Azerbaijan’s Ministry of Digital Development and Transport, in collaboration with ITU, conducted a comprehensive digital skills assessment of 35,000 individuals using a randomized sample of households in 13 regions for the first time using ITU’s new Digital Skills Indicator. The study was launched in Baku in May 2024. In the same month, Romania’s Ministry of Research, Innovation and Digitization, with support of the World Bank, launched inter-ministerial consultations to create the National Digital Skills Framework (DigCompRo) based on the DigComp framework.

Another effort being made is to eliminate socio-economic biases from existing measures. For example, the Programme for the International Assessment of Adult Competencies (PIAAC) offers promising avenues for comparative analysis across gender, age groups, education levels, and migration backgrounds. The emphasis on problem-solving in technology-rich environments, as exemplified by PIAAC’s module, underscores the importance of aligning measurement methodologies with evolving societal needs.

Updating existing measures to integrate new technologies is another critical focus area. With the increasing importance of coding and programming skills and their intersection with digital skills, the International Computer and Information Literacy Study (ICILS) has developed a separate measure of coding skills. New modules introduce tasks such as developing, testing, fixing errors in computer codes, running simulations, and emphasizing real-life applications such as game development.

Similarly, the 2025 Programme for International Student Assessment (PISA), with a focus on Learning in the Digital World, will not only examine students’ digital skills but also assess self-regulated learning in an open-ended digital environment. Students will conduct experiments, analyse data, and build and debug computational artefacts. They will have access to various resources and receive feedback. Moreover, their performance will be represented by a computational model, analysing response and process data.

New data are being collected. In the United States, the Digital Equity Capacity Grants, established under the 2021 Digital Equity Act, present a unique opportunity for states to allocate funds efficiently and harness the opportunity for comprehensive data collection on digital skills. With artificial intelligence transforming many areas of life, the National AI Research and Development Strategic Plan 2023 Update allocates funding to raise AI-related skills. It will be expected to develop ways of measuring AI-related skills too.

Measuring teachers’ digital skills is also important. The Digital Competence Framework for Educators (DigCompEdu) has been developed specifically for this purpose. In Southeastern Europe, the Digital Needs Analysis for Teachers Tool, based on the DigCompEdu framework, has been utilized in countries such as Albania and Serbia. This tool allows teachers to self-assess their digital competencies, offering insights into specific needs by country and school type. Its application has facilitated targeted professional development and enabled policymakers to analyse educational needs and the influence of data on educational policy, particularly in teacher education. However, objective measures of teachers’ digital skills remain rare. Two new focus areas for ICILS 2023 are teaching practices and teachers’ beliefs and leadership for ICT.

The accurate measurement of digital skills is vital for enhancing education, promoting equitable access to technology, and ensuring everyone possesses the skills needed to thrive in the rapidly evolving digital world. By addressing inequalities and staying abreast of technological advancements, we ensure that these measurements are not only precise but also fair and beneficial.  



Source link