Taking a future perspective by learning from the past – A systematic review of assessment instruments that aim to measure primary and secondary school students' ICT literacy

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There seems to be a new wave coming, concerned with even broader skillsets such as 21st-century skills, in which ICT literacy is often described as a subset.

“There seems to be a new wave coming, concerned with even broader skillsets, such as 21stcentury skills (Voogt & Roblin, 2012), and in which ICT literacy is often described as a subset (Griffin et al., 2012).”

• cites Voogt & Roblin, 2012 sourceSupported
• cites Griffin et al., 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

There are two specific approaches to measuring ICT literacy: one assumes ICT is closely related to subject content, and another presumes ICT literacy transcends individual disciplines and comprises transferable knowledge, skills, and understandings.

“There are two specific approaches to measuring ICT literacy, one which assumes that ICT is closely related to subject content (e.g., mathematics, science, reading) and another which presumes that ICT literacy transcends individual disciplines and comprises a set of knowledge, skills, and understandings that learners can adapt and transfer to new contexts (Fraillon, Schulz, & Ainley, 2013).”

• cites Fraillon, Schulz, & Ainley, 2013Supported

Probably not a substantial claim: may not pass S3 (load-bearing).

From an economic rationalist view in which students' success is strongly linked to the outcomes of education, trustworthy outcomes from assessments are required.

“Also, from an economic rationalist view in which students' success is strongly linked to the outcomes of education, trustworthy outcomes from assessments are required (MacDonald & Brooker, 1999;Newton & Shaw, 2014).”

• cites MacDonald & Brooker, 1999Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Newton & Shaw, 2014 sourceCould not verify

  The available passages, headed “Assessment for Learning” (page not given), discuss assessment’s role in learning, grading, and feedback. Searches for economic rationalism and trustworthy assessment outcomes found no passage linking students’ success to educational outcomes in the way claimed. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

ICT literacy is vital for young people.

“ICT literacy has been pointed out as vital for young people”

• cites Boyd, 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Helsper & Eynon, 2010 sourcePartly supported

  In the 'Additional information' section (page not provided), the paper says that young people's interactions with technology affect what they can learn in school and links breadth of technology use to confidence in ICT skills and digital learning. This supports the importance of ICT skills, but the paper does not establish the stronger claim that ICT literacy is vital for young people. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites Selwyn, 2009 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Students lack necessary ICT competences.

“it has been identified that students lack necessary ICT competences”

• cites Boyd, 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Helsper & Eynon, 2010 sourcePartly supported

  The Discussion (page unavailable) warns that teachers may assume a level of technological knowledge that is not accurate for all students. It also says that using the Internet for information does not necessarily mean young people can critically assess it. These passages support gaps in some students’ skills, but not the unqualified claim that students generally lack necessary ICT competences. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Selwyn, 2009 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Newton and Shaw (2014) suggested a new framework for 21st-century evaluation.

“Newton and Shaw (2014) have made an attempt to address this and suggested a new framework for 21st-century evaluation.”

• cites Newton and Shaw (2014) sourceCould not verify

  The indexed copy contains an Assessment Reform Group leaflet on principles of assessment for learning, with no section or page labels. Searches for a “new framework for 21st-century evaluation” and for Newton and Shaw’s evaluation framework found no passage addressing the claimed proposal. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

May combine two claims.

Similar descriptions to DIGCOMP's Technical Operational Skills area can be found in the CIL framework, but this is not explicitly mentioned in the 21st-century skills framework.

“Finally, for the area Technical Operational Skills, similar descriptions can be found in the CIL framework (e.g., Knowing about and Understanding Computer Use [1.1]), but this is not explicitly mentioned in the 21st-century skills framework.”

May combine two claims.

In general, the three frameworks emphasise students' ability to collect and understand information; to produce information; to communicate digital information; and to search, produce, and communicate in a safe and responsible way.

“In general, the three frameworks emphasise students' ability to collect and understand information; to produce information; to communicate digital information; and to search, produce, and communicate in a safe and responsible way.”

May combine two claims.

Similar descriptions to DIGCOMP's Problem Solving area can be found in the 21st-century skills framework category 2, but not in the CIL framework.

“Similar descriptions can be found in the 21st-century skills framework category 2, but not in the CIL framework.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The collaboration aspect of communication is lacking in the CIL framework.

“Note that the collaboration aspect of communication is lacking in the CIL framework.”

May combine two claims.

An initial literature search revealed limited access to handbooks and systematic reviews considering the research area, yet a number of primary research studies exist.

“An initial literature search revealed limited access to handbooks and systematic reviews considering the research area described above, yet a number of primary research studies exist.”

May combine two claims.

The hypothesized three-factor structure of TEDDICS was supported by the ESEM and showed improved model fit compared to a CFA model.

“The results indicate that the hypothesized three-factor structure of TEDDICS was supported by the ESEM and showed improved model fit compared to a CFA model.”

May combine two claims.

A lack of tests that measured Synchronous Digital Communication and Collaboration, Safety, Programming, or Problem Solving was identified.

“In particular, a lack of tests that measured Synchronous Digital Communication and Collaboration, Safety, Programming, or Problem Solving was identified.”

May combine two claims.

With regard to Terenzini's taxonomy, the ICT literacy assessments relate to the category Campus and Program Evaluation and not to the category Gatekeeping Tests at an Individual Level.

“Moreover, with regard to Terenzini's taxonomy, the ICT literacy assessments relate to the category Campus and Program Evaluation and not to the category Gatekeeping Tests at an Individual Level (Terenzini, 1989).”

• cites Terenzini, 1989Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The five level descriptors in each category in the Norwegian ICT literacy framework do not correspond with the grades in the Norwegian educational system, and further explanations regarding this link are not provided in the framework.

“Moreover, the five level descriptors in each category in the Norwegian ICT literacy framework do not correspond with the grades in the Norwegian educational system (i.e., 13 years of compulsory education), and further explanations regarding this link are not provided in the framework (see Norwegian Directorate for Education and Training, 2012).”

• cites Norwegian Directorate for Education and Training, 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The ESEM model showed better fit and confirmed the assumptions about the three distinct but related factors.

“Moreover, the ESEM model showed better fit and confirmed the assumptions about the three distinct but related factors.”

May combine two claims.

The majority of the tests reported fewer than ten indicators, suggesting that the reported information about the quality of the instruments and their intended interpretations was by and large incomplete across the reviewed papers.

“The majority of the tests reported fewer than ten indicators, suggesting that the reported information about the quality of the instruments and their intended interpretations was by and large incomplete across the reviewed papers.”

May combine two claims.

The largest difference between the DIGCOMP framework and the Norwegian curriculum is that the Norwegian curriculum is less detailed, and many of the single competences in DIGCOMP are included in the level descriptions instead.

“The largest difference between the two is that the Norwegian curriculum is less detailed, and many of the single competences in DIGCOMP are included in the level descriptions instead (see Appendix A for an overview of the Norwegian ICT literacy framework).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The structure of the Norwegian ICT literacy framework is somewhat misleading.

“the structure of the Norwegian framework is somewhat misleading.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Norwegian teachers perceive ICT literacy as important only for students in the beginning of primary education, rather than as skills continuously developed throughout education.

“the teachers perceive it as important only for students in the beginning of primary education, instead of skills that are continuously developed as part of their subject domain learning throughout their educational training”

• cites Aasen et al., 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The hypothesized four-dimensional Rasch model fit the data better than a unidimensional Rasch model, which indicates that the underlying framework could be operationalized.

“The results showed that the hypothesized four-dimensional Rasch model fit the data better than a unidimensional Rasch model, which indicates that the underlying framework could be operationalized.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The empirical investigations of the LDN-ICT test revealed that a Rasch model with four dimensions had a significantly improved model fit as compared to a unidimensional model, lending support to the theoretical framework.

“The empirical investigations of the LDN-ICT test revealed that a Rasch model with four dimensions had a significantly improved model fit as compared to a unidimensional model, lending support to the theoretical framework.”

May combine two claims.

Five items which indicated DIF across students' gender and SES were removed, and evidence for an invariant construct was ensured.

“Five items which indicated DIF across students' gender and SES were removed, and evidence for an invariant construct was ensured.”

Probably not a substantial claim: may not pass S3 (load-bearing).

How ICT literacy is understood varies across Norwegian schools and classrooms.

“An evaluation of the Norwegian national curriculum revealed that how ICT literacy is understood varies across schools and classrooms”

• cites Aasen et al., 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The LDN-ICT test operationalized competences within the competence areas Information and Content Creation, which are well covered as suggested by the systematic review findings.

“The LDN-ICT test operationalized competences within the competence areas Information and Content Creation, which are well covered as suggested by the systematic review findings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A categorisation level that could bridge the topical content and the ability level descriptors is missing in the Norwegian ICT literacy framework.

“One categorisation level, which could potentially bridge the topical content (i.e., labeled as categories in the framework) and the ability level descriptors in the framework is missing”

May combine two claims.

The empirical investigations of the TEDDICS scale (Paper 2) revealed that the underlying framework could be identified, and a multidimensional model consisting of the three factors fit better than a unidimensional model.

“Nevertheless, the empirical investigations of the TEDDICS scale (Paper 2) revealed that the underlying framework could be identified, and a multidimensional model consisting of the three factors fit better than a unidimensional model.”

May combine two claims.

Positive and statistically significant correlations between the four dimensions of LDN-ICT and ICT self-efficacy and academic aspirations were revealed, indicating evidence for the test's external validity.

“Positive and statistically significant correlations between the four dimensions of LDN-ICT and ICT self-efficacy and academic aspirations were revealed, indicating evidence for the tests' external validity.”

May combine two claims.

Positive correlations between SES and the four dimensions of LDN-ICT were revealed, yet no gender differences were indicated.

“Thus, further comparisons across these groups could be made, and positive correlations between SES and the four dimensions were revealed, yet no gender differences were indicated.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A fairly low level of emphasis on evaluation of digital information was indicated.

“In contrast, a fairly low level of emphasis on evaluation of digital information was indicated.”

May combine two claims.

Evidence for the highest level of invariance (strict invariance) was ensured, and consequently comparisons of the factor means between male and female teachers and across the different main subjects could be employed.

“Moreover, evidence for the highest level of invariance (strict invariance) was ensured, and consequently comparisons of the factor means between male and female teachers and across the different main subjects could be employed.”

May combine two claims.

The LDN-ICT test measures competences within the competence areas Communication and Problem Solving, which have been scarcely operationalized in previous studies.

“In addition, the LDN-ICT test measures competences within the competence areas Communication and Problem Solving, which have been scarcely operationalized in previous studies (Quellmalz, 2009).”

• cites Quellmalz, 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Even though further refinements of the TEDDICS and LDN-ICT scales are recommended (e.g., need for more items in some sub-dimensions), the studies provide statistical evidence for the multidimensionality of the scales aligned with the underlying frameworks.

“In sum, even though further refinements of the TEDDICS and LDN-ICT scales are recommended (e.g., need for more items in some sub-dimensions), the studies provide statistical evidence for the multidimensionality of the scales aligned with the underlying frameworks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The findings of the TEDDICS study revealed that teachers to a great extent emphasize developing students' competence in searching for digital information.

“The findings of the TEDDICS study revealed that teachers to a great extent emphasize developing students' competence in searching for digital information.”

May combine two claims.

The competence areas Content Creation, Communication, and Technical Operational Skills are included to some extent, whereas few tests cover the competence areas Safety and Problem Solving.

“The competence areas Content Creation, Communication, and Technical Operational Skills are included to some extent, whereas few tests cover the competence areas Safety and Problem Solving (see Table 1 for an overview of the competence areas and competences).”

May combine two claims.

Further analysis showed significant mean differences in TEDDICS in favor of teachers in the humanities, language, and art, yet no significant gender differences were found.

“Further analysis showed significant mean differences in TEDDICS in favor of teachers in the humanities, language, and art, yet no significant gender differences were found.”

May combine two claims.

The correlations between the LDN-ICT test and collective efficacy and perceived usefulness of ICT were positive, nevertheless insignificant.

“The correlations between the LDN-ICT test and collective efficacy and perceived usefulness of ICT were positive, nevertheless insignificant.”

May combine two claims.

ICILS was conducted in 22 countries and operationalized parts of the ICT literacy framework, particularly emphasizing the competence area Information.

“ICILS was conducted in 22 countries and operationalized parts of the ICT literacy framework, particularly emphasizing the competence area Information.”

Probably not a substantial claim: may not pass S4 (based).

The findings of the LDN-ICT paper provide evidence for the validity and reliability of the test.

“The findings of the LDN-ICT paper provide evidence for the validity and reliability of the test”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The novelty of the Newton and Shaw framework and how it addresses the challenges of 21st-century assessment are not clear.

“the novelty of the framework and how it addresses the challenges of 21st-century assessment are not clear.”

May combine two claims.

The reporting of what was measured in the tests varied, and in some cases it was briefly reported that the test measured ICT literacy at the framework level or information and communication at the competence-area level without further detailed explanation.

“the reporting of what was measured in the tests varied, and in some cases it was briefly reported that the test measured, for instance, ICT literacy (the framework level) or information and communication (the level of competence area), without further detailed explanation.”

May combine two claims.

ICT literacy assessments vary to a large degree in both content covered and the quality with which they were reported.

“the ICT literacy assessments vary to a large degree, both regarding content covered and the quality with which they were reported.”

Classified as probably not a substantial claim. May combine two claims.

The Newton and Shaw framework is fairly general, which makes it challenging to apply for appraising the reported quality of tests.

“the framework is fairly general, which makes it challenging to apply it for appraising the reported quality of tests.”

Classified as probably not a substantial claim.

Reliability and validity evidence stood out as an important aspect for evaluating the quality of assessments.

“The reliability and validity evidence stood out as an important aspect for the evaluation of the quality of assessments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No significant gender differences were indicated in achievement on the LDN-ICT test.

“No significant gender differences were indicated in achievement on the LDN-ICT test.”

Classified as probably not a substantial claim.

Two indicators were developed to address some of the peculiarities of ICT literacy tests.

“two indicators were developed to address some of the peculiarities of ICT literacy tests.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The few indicators not reported for TEDDICS and LDN-ICT were mainly seen as irrelevant given the purpose of the instruments.

“The few indicators not reported for these two instruments were mainly seen as irrelevant given the purpose of the instruments.”

May combine two claims.

There are positive correlations between students' test scores, SES, and academic aspirations.

“The positive correlations between the students' test scores, SES, and academic aspirations support previous results”

The quote only partly states this claim.

The digital divide seems to be prevalent also with regard to students' learning in digital networks, including Information Search, Retrieval, and Evaluation; Communication; and Collaborative problem-solving.

“the digital divide pointed out in previous literature seems to be prevalent also with regard to students' learning in digital networks”

Classified as probably not a substantial claim.

Because reports from the assessments referenced a spectrum of validity literature or none, it was not considered useful to impose one particular theoretical position in reviewing their quality.

“Given that the reports from these assessments gave reference to a spectrum of validity literature (or none), it was not considered useful to impose one particular theoretical position in reviewing their quality.”

May combine two claims.

TEDDICS reported 12 reliability and validity indicators, and LDN_ICT reported 14 indicators.

“TEDDICS reported 12 indicators; LDN_ICT reported 14 indicators”

Classified as probably not a substantial claim.

Four categories were developed for capturing the authenticity and innovativeness of task/item designs, based on frameworks for constructing innovative item types.

“four categories were developed on the basis of frameworks for constructing innovative item types”

• cites Parshall, Harmes, Davey, & Pashley, 2010Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Scalise & Gifford, 2006Not checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Paper 3 serves as an example of a validation study that successfully reports vital aspects of the indicators.

“Paper 3 serves as an example of a validation study which successfully manages to report vital aspects of the indicators.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The previous reporting of the scales and the two papers in the dissertation successfully demonstrated high levels of transparency in reporting what was measured and how.

“the previous reporting of the scales and the two papers in the dissertation successfully demonstrated high levels of transparency in the reporting of what was measured and how”

Probably not a substantial claim: may not pass S4 (based).

No prior empirical study in this field has investigated the external validity of an assessment that includes students' synchronous communication and collaborative problem solving.

“No prior empirical study in this field has investigated the external validity of an assessment which includes students' synchronous communication and collaborative problem solving.”

Probably not a substantial claim: may not pass S2 (checkable).

The dissertation can be considered a contribution to the literature and the field of ICT literacy, test development, and assessment.

“this dissertation can be considered a contribution to the literature and the field of ICT literacy, test development, and assessment”

Probably not a substantial claim: may not pass S2 (checkable).

The revised DIGCOMP has functioned satisfactorily as a reference framework in the dissertation.

“it has functioned satisfactorily as a reference framework in the dissertation.”

Classified as probably not a substantial claim.

The LDN-ICT test's Social Capital and Intellectual Capital dimensions included few items.

“more items should be added to the two dimensions which included few items (i.e., Social Capital and Intellectual Capital).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The content of the LDN-ICT test mimics to a large degree real-world tasks by including interactivity, open access to the Internet, and freely available tools for communication and collaboration.

“The content of the test mimics to a large degree real-world tasks by including interactivity, open access to the Internet, and freely available tools for communication and collaboration.”

Probably not a substantial claim: may not pass S4 (based).

Comparing two relatively well-known approaches based on distinct assumptions about the theoretical underpinnings of the study revealed the strengths and weaknesses of the construct under investigation.

“In particular, comparing two relatively well-known approaches which are based on distinct assumptions about the theoretical underpinnings of the study revealed the strengths and weaknesses of the construct under investigation.”

May combine two claims.

The use of LinkedIn and ResearchGate for detecting ICT literacy tests was helpful and identified tests that otherwise would not have been captured by database searches.

“the use of LinkedIn and ResearchGate for detecting ICT literacy tests was helpful, and tests that otherwise would not have been captured by the database searches were identified.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The indicators developed in the systematic review are by no means exhaustive and should be investigated further.

“These are by no means exhaustive and should be investigated further.”

Probably not a substantial claim: may not pass S2 (checkable).

The systematic review approach taken in the dissertation contributes to the field.

“the systematic review approach taken in this dissertation contributes to the field.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The revised DIGCOMP has been proven to have the necessary breadth and depth to cover competence areas and levels.

“it has been proven to have the necessary breadth and depth to cover competence areas and levels”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The dissertation can be considered an original empirical contribution to the literature in the field of ICT literacy, test development, and assessment.

“this dissertation can be considered an original empirical contribution to the literature in the field of ICT literacy, test development, and assessment.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The dissertation contributes to knowledge on ICT literacy assessment in education.

“The work presented in the dissertation contributes to the knowledge on ICT literacy assessment in education.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most terms related to ICT knowledge are used interchangeably and reflect the same content to a great extent.

“However, it seems to be a challenging task, and many researchers have concluded that most of the terms are used interchangeably and reflect the same content to a great extent (Law et al., 2009;Søby, 2013).”

• cites Law et al., 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Søby, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Such concepts are most often accompanied with frameworks that outline and detail the specific content related to the concept.

“Thus, such concepts are most often accompanied with frameworks, which outline and detail the specific content related to the concept.”

May combine two claims.

Self-reports have been shown to be biased, for example toward gender, and may not provide an accurate picture of students' actual competence.

“In this study, the latter assessment approach was selected, because self-reports have been shown to be biased (e.g., toward gender [Aesaert et al., 2014;Hakkarainen et al., 2000]) and may not provide an accurate picture of students' actual competence.”

• cites Aesaert et al., 2014 sourceNot checked — no full text of the source
• cites Hakkarainen et al., 2000 sourceNot checked — no full text of the source

May combine two claims.

Students lack essential skills within the ICT literacy framework, including information retrieval and processing, evaluating information, and communicating and collaborating in digital environments.

“Research has demonstrated that students lack essential skills within the ICT literacy framework.For instance, Strømsø and Bråten (2014) showed that students lacked skills related to information retrieval and processing.”

• Also stated in: § 1.1 Background and Rationale · plain

  “These findings were supported by several studies that identified gaps in students' competences in evaluating information (Aesaert, van Nijlen, Vanderlinde & van Braak, 2014;Fang, 2012), communicating and collaborating in digital environments (Calvani, Fini, Ranieri & Picci, 2012;Kuiper, Volman & Terwel, 2005).”

• cites Strømsø and Bråten (2014)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Aesaert et al., 2014 sourceNot checked — no full text of the source
• cites Calvani et al., 2012 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Teachers' ICT use varies considerably; ICT is used most frequently for relatively simple tasks and less for more complex tasks.

“A report from the International Computer and Information Literacy study (ICILS) revealed that teachers' ICT use varied considerably; ICT was used most frequently for relatively simple tasks (e.g., word processing, presentations, information search and retrieval) and less for more complex tasks (e.g., enabling student collaboration, assessment, and feedback) (Fraillon et al., 2014).”

• cites Fraillon et al., 2014 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Concepts of ICT literacy do not reflect what it means to be ICT literate or which specific skills, attitudes, and competences students should attain.

“Nevertheless, the concepts themselves do not reflect, for instance, what it is meant by being ICT literate, or which specific skills, attitudes, and competences students should attain.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All studies in the systematic review were coded by two independent researchers.

“Data from each study were extracted by following a coding scheme, and all studies were coded by two independent researchers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A change in ICT may not only affect ICT literacy, but also induce changes in more generic cognitive or non-cognitive competences.

“Thus, a change in ICT may not only affect ICT literacy, but also induce changes in more generic cognitive or non-cognitive competences.”

Classified as probably not a substantial claim.

A comprehensive understanding of ICT literacy policies and practices in specific contexts requires investigating all three domains: the intended, implemented, and attained curriculum.

“Markauskaite (2006) emphasised that “in order to develop a comprehensive understanding of ICT literacy policies and practices in specific contexts, all three domains should be investigated” (p. 6).”

• cites Markauskaite (2006)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Different terms used synonymously for describing ICT-related capabilities do not necessarily convey exactly the same meaning.

“Markauskaite (2006) argued that even though different terms are used synonymously for describing ICT-related capabilities, they do not necessarily convey exactly the same meaning.”

• cites Markauskaite (2006)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Teachers' ICT self-efficacy and perceived usefulness of ICT are strongly related to their actual integration of ICT in learning environments.

“Moreover, research has shown that teachers' ICT self-efficacy and perceived usefulness of ICT are strongly related to their actual integration of ICT in learning environments (Chien, Wu, & Hsu, 2014;Scherer, Siddiq, & Teo, 2015).”

• cites Chien, Wu, & Hsu, 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Scherer, Siddiq, & Teo, 2015 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Most scholars agree that operationalization of ICT literacy has changed over time due to the advancement and changes in technology.

“Most scholars agree that operationalization of ICT literacy has changed over time due to the advancement and changes in technology (Erstad, 2006).”

• cites Erstad, 2006 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The three frameworks (DIGCOMP, CIL, and 21st-century skills) convey many of the same principles, are all related to education, and aim to describe what and how students acquire, use, adapt to, and learn with technology.

“Although the three frameworks were developed within different larger projects for rather different purposes, they convey many of the same principles, are all related to education, and, moreover, aim to describe what and how, students acquire, use, adapt to, and learn with technology.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The definition of ICT literacy used in this thesis is in line with several other definitions of ICT literacy and digital competence that reflect the importance of confident and critical use of ICT for fully participating in the knowledge society.

“This definition is in line with several other definitions of ICT literacy (Educational Testing Service [ETS], 2007;Ministerial Council on Education, Employment, Training and Youth Affairs [MCEETYA], 2005) and digital competence (Ferrari, 2013) which reflect the importance of confident and critical use of ICT for fully participating in the knowledge society.”

• cites ETS, 2007Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites MCEETYA, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ferrari, 2013 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Norway was one of the first countries to enhance the status of ICT literacy by including it in the national curriculum.

“Norway was one of the first countries to enhance the status of ICT literacy by including it in the national curriculum (Balanskat & Gertsch, 2010;Krumsvik, 2008), and it is the teachers who are formally responsible for teaching ICT literacy.”

• cites Balanskat & Gertsch, 2010Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Krumsvik, 2008 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The evolution of technology and society has been a driver for continuous change of the concepts and their content.

“She further argued that the evolution of technology and society has been a driver for continuous change of the concepts and their content.”

• cites Markauskaite (2006)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In Norway, teachers are formally responsible for teaching ICT literacy.

“Norway was one of the first countries to enhance the status of ICT literacy by including it in the national curriculum (Balanskat & Gertsch, 2010;Krumsvik, 2008), and it is the teachers who are formally responsible for teaching ICT literacy.”

• cites Balanskat & Gertsch, 2010Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Krumsvik, 2008 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Educational systems are now challenged to prepare young people for a future with uncertain job demands.

“As a consequence, educational systems are now challenged to prepare young people for a future with uncertain job demands (Dede, 2009).”

• cites Dede, 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a strong connection between ICT-related competences and other capabilities or literacies.

“The literature also highlights the strong connection between ICT-related competences and other capabilities or literacies (e.g., numeracy, reading and science literacy, creativity, productivity, communication, collaboration skills; North Central Regional Educational Laboratory [NCREL], 2003).”

• cites NCREL, 2003Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The Norwegian ICT literacy framework has some limitations compared with the international frameworks.

“Yet, in comparison, the VII Norwegian ICT literacy framework has some limitations; suggestions for further revisions are given.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Focus-group interviews were administered to collect information about students' perceptions and experiences with sitting for the test.

“Furthermore, focus-group interviews were administered to collect information about students' perceptions and experiences with sitting for the test.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Messick (1995) emphasized a unified concept of construct validity that integrates the traditional single conceptions, namely content, criterion, and construct validities.

“This was explicitly argued by Messick (1995), as he emphasized a unified concept of construct validity that integrates the traditional single conceptions, namely content, criterion, and construct validities.”

• cites Messick, 1995 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

From a test development perspective, one of the key aims is to develop measures with evidence to reflect the intended construct to a large degree.

“This is also accentuated from a test development perspective, in which one of the key aims is to develop measures with evidence to reflect the intended construct to a large degree.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Both validity and reliability are generic meta-concepts, and in discussing the validity and reliability of results and claims from a specific study, the concept needs to be contextualized to take into account the research design, the methods used, and the overall purpose of the study.

“Moreover, both validity and reliability are generic meta-concepts, and in discussing the validity and reliability of the results and claims made from a specific study, the concept needs to be contextualized to take into account the research design, the methods used, and the overall purpose of the study.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For each category in the Norwegian ICT literacy framework, descriptions of progression through five levels are provided, and each competence category formulates performance standards at that level.

“For each of these categories, descriptions of the progression through five levels are provided, and each competence category (i.e., the cells) in the grid formulates performance standards at that level”

Classified as probably not a substantial claim. May combine two claims.

In establishing a validity argument for the LDN-ICT study, it was vital to ensure that the revised LDN-ICT test measures the intended construct, which was done with different approaches (translation and revisions involving different people, think-aloud protocols, interviews, and a pilot).

“In establishing a validity argument for this study, it was vital to ensure that the revised LDN-ICT test measures the intended construct, which was done with different approaches (i.e., translation and revisions of the original test by involving different people, think-aloud protocols, interviews, and a pilot).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Norwegian ICT literacy framework is outlined in a grid and consists of four categories: Search and Process, Produce, Communicate, and Digital Judgment.

“The ICT literacy framework is outlined in a grid and consists of four categories, namely: Search and Process, Produce, Communicate, and Digital Judgment.”

May combine two claims.

The reporting of the quality of the tests and test development process was investigated against 15 indicators, supporting the reliability evidence and the creation of a validity argument.

“Furthermore, the reporting of the quality of the tests and test development process was investigated against 15 indicators, supporting the reliability evidence and the creation of a validity argument.”

Classified as probably not a substantial claim.

Another source of diversity between existing ICT literacy assessments is the degree to which the tests capture actually performed competence, i.e., the degree to which tests are based on authentic tasks conducted in test administration software representing the dynamic and interactive nature of regular software.

“Another source of diversity between existing ICT literacy assessments is the degree to which the tests capture actually performed competence, in other words the degree to which the tests are based on authentic tasks conducted in test administration software representing the dynamic and interactive nature of regular software.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Given the flexibility of EFA, latent variables corrected for measurement error cannot be incorporated, and therefore constructs identified through EFA have to be converted to suboptimal scale or factor scores.

“Moreover, given the flexibility of EFA, latent variables corrected for measurement error cannot be incorporated, and therefore constructs identified through EFA have to be converted to suboptimal scale or factor scores (Marsh et al., 2014).”

• cites Marsh et al., 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The results and inferences of the three studies in the dissertation were discussed against existing theory and with researchers in the field (e.g., at conferences).

“With regard to this, the results and inferences of the three studies in the dissertation were discussed against existing theory and with researchers in the field (e.g., at conferences).”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is challenging to incorporate latent variables that follow a structure with cross-loadings (derived from EFA) into subsequent analyses such as testing relations with other constructs.

“Additionally, the researchers argued that it is challenging to incorporate latent variables that follow a structure with cross-loadings (derived from EFA) into subsequent analyses such as testing relations with other constructs.”

• cites Marsh et al., 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

After the pilot, the test data were used to statistically investigate the reliability and validity of the test, as reported in Paper 3.

“After the pilot, the test data were used to statistically investigate the reliability and validity of the test, which is reported in Paper 3.”

Classified as probably not a substantial claim. May combine two claims.

Petticrew and Roberts (2006) provided five arguments for when one should conduct a systematic review; three were particularly relevant for Paper 1: when there is a wide range of research but key questions remain unanswered, when a general overall picture is needed to direct future research, or when an accurate picture of past research is required to promote new methodologies.

“Petticrew and Roberts (2006) provided five arguments for when one should conduct a systematic review.Three of them were particularly relevant for Paper 1: (1) when it is known that there is a wide range of research on a subject but where key questions remain unanswered, (2) when a general overall picture of the evidence in a topic area is needed to direct future research efforts, or (3) when an”

• cites Petticrew and Roberts (2006)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The process of assessment development is not linear but cyclic, emphasizing an iterative process.

“The process of assessment development is not linear as described here, yet cyclic, emphasizing an iterative process (Wilson, 2005).”

• cites Wilson, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Administering the English version of the LDN-ICT test to Norwegian students would likely yield biased results favoring students with higher English proficiency and introduce a major source of construct-irrelevant variance.

“Nevertheless, the chances for getting biased results were high given that the test probably would favor students with higher proficiency in English, or stated differently, a test in English would have introduced a major source of construct irrelevant variance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A test must be reliable to be valid, but the opposite does not apply, and therefore reliability is considered one facet of validity.

“Hence, they concluded that a test must be reliable to be valid, but the opposite does not apply, and therefore reliability is considered one facet of validity.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Reliability and validity investigations play a vital role as a process of scrutinizing the quality of assessments.

“This leads to the vital role of reliability and validity investigations as a process of scrutinizing the quality of assessments.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Assessment is closely related to teaching and learning, and the three are inextricably linked, as each informs the others.

“It has been argued that assessment is closely related to teaching and learning, and that the three are inextricably linked, as each informs the others (Black, 1999;Wilson, 2005).”

• cites Black, 1999Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wilson, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

These theoretical perspectives share common ground regarding their view on validity as a process and not a state per se-it is not a test that is validated.

“Although these theoretical perspectives differ to some extent, they share common ground regarding their view on validity as a process and not a state per se-it is not a test that is validated.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

There is a risk that by using different terminology, the existing literature may not be fully connected.

“Also, there is a risk that by using different terminology, the existing literature may not be fully connected.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

On the basis of the initial coding, the coding scheme was further developed and revised and used across all included studies to extract the relevant information.

“On the basis of the initial coding, the coding scheme was further developed and revised and used across all included studies to extract the relevant information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Paper 1, several processes were followed as suggested in many systematic review methodology guidelines.

“In Paper 1, several processes were followed as suggested in many systematic review methodology guidelines (Gough et al., 2012;Petticrew & Roberts, 2006).”

• cites Gough et al., 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Petticrew & Roberts, 2006Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ICT literacy is not a subject in compulsory education in Norway, but it is defined as one of the basic key literacies to be integrated with the competence aims of the school subjects.

“ICT literacy is not a subject in the compulsory education, but it is defined as one of the basic key literacies (along with reading, writing, oral skills, and numeracy) to be integrated with the competence aims of the school subjects (Norwegian Directorate for Education and Training, 2012).”

• cites Norwegian Directorate for Education and Training, 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In IRT, the estimation of respondents' abilities is independent of the actual items included in the test and vice versa.

“Conversely, in IRT, the estimation of respondents' abilities is independent of the actual items included in the test and vice versa (Wilson, 2005).”

• cites Wilson, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In Paper 1, the information was extracted following a coding scheme which was developed in an iterative process.

“In Paper 1, the information was extracted following a coding scheme which was developed in an iterative process.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The peer-review process in academia contributed to the credibility of the research (e.g., conference presentation proposals and journal submissions).

“In addition, the peer-review process in academia contributed to this (e.g., conference presentation proposals and journal submissions).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Trustworthy measures are particularly critical from a social justice perspective; principles of fairness and equity emphasize that students should be provided with equal opportunities within and across classrooms and schools.

“Trustworthy measures are particularly critical from a social justice perspective; principles of fairness and equity emphasize that students should be provided with equal opportunities within and across classrooms and schools (OECD, 2001).”

• cites OECD, 2001Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

An evaluation of the categories of the integrative framework justifies placing the first two studies (Systematic review and TEDDICS) at the right side of the scale, indicating findings/knowledge at the high-inference level.

“An evaluation of these categories justifies placing the first two studies (Systematic review and TEDDICS) at the right side of the scale, indicating findings/knowledge at the high-inference level.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In assessment, reliability is often seen as an integral part of validity.

“In assessment, reliability is often seen as an integral part of validity (Wilson, 2005).”

• cites Wilson, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Messick's (1995) definition of validity indicates that all phases of the assessment development framework are important for establishing an argument for how well the measures derived from an assessment reflect the construct of interest.

“This definition indicates that all phases of the assessment development framework, for instance, are important for establishing an argument for how well the measures derived from an assessment reflect the construct of interest.”

• cites Messick, 1995 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Paper 3 is closer to the middle of the scale because it had a small sample size and the students were purposefully sampled.

“Paper 3, on the other hand, is closer to the middle of the scale because it had a small sample size and the students were purposefully sampled.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Item fit, local independence, and differential item functioning (DIF) are useful approaches to investigate the validity of a scale.

“Specifically, item fit (i.e., measure of the discrepancy between the observed and theoretical scores; Wu & Adams, 2013), local independence (the responses to the items are independent; Smith, 2002), and differential item functioning (DIF) (i.e., measurement bias for an item that occurs when different subgroups (e.g., gender) in a sample have different probability of giving a correct response given”

• cites Wu & Adams, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Smith, 2002Not checked — no full text of the source
• cites Tennant & Pallant, 2007Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The ICT literacy term is broad and consists of several competence areas and competences and may not be easily measured in one test.

“In addition to these mere general views, several reflections with regard to ICT literacy assessment came into play, such as the comprehensiveness of the ICT literacy term, which is broad and consists of several competence areas and competences (Calvani, Cartelli, Fini, & Ranieri, 2008;Eshet-Alkalai, & Amichai-Hamburger, 2004;Ferrari, 2013) and which may not be easily measured in one test (Calvani”

• cites Calvani et al., 2008Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Eshet-Alkalai, & Amichai-Hamburger, 2004 sourceNot checked — no full text of the source
• cites Ferrari, 2013 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Several steps were taken to ensure the validity and reliability of the LDN-ICT test.

“Several steps were taken to ensure the validity and reliability of the test.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the quality appraisal phase, a considerable amount of theoretical work was done to generate the criteria for appraising the quality of the studies included in the review.

“For this phase, a considerable amount of theoretical work was done to generate the criteria for appraising the quality of the studies included in the review.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

After several rounds of translation and revisions, students' think-aloud protocols while taking the test were used to ensure that students understood the tasks as intended and that the software functioned properly.

“First, after several rounds of translation and revisions, students' think-aloud protocols while they were taking the test were used to ensure that the students understood the tasks as they were intended and that the software functioned properly.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The benefits of IRT are particularly important for the transferability and reuse of assessment instruments and allow designing instruments with specific characteristics.

“Furthermore, it has been accentuated that these benefits are particularly important with regard to the transferability and reuse of assessment instruments, and they render the possibility to design instruments with specific characteristics (de Ayala, 2013).”

• cites de Ayala, 2013 sourceNot checked — no full text of the source

May combine two claims.

Students at higher ICT literacy levels in the Norwegian framework are expected to manage evaluation of information and use proper search strategies, whereas students at lower levels are not expected to be able to search for or assess information.

“This indicates that students at higher ICT literacy levels are expected to manage evaluation of information and use proper search strategies, whereas students at lower levels are not expected to be able to search for or assess information.”

May combine two claims.

This finding is critical and questions the trustworthiness of the tests and the credibility of the results acquired with a specific test.

“This finding is critical and questions the trustworthiness of the tests and the credibility of the results acquired with a specific test.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The importance of reliable and valid measures for supporting teaching and learning is emphasized, and assessment is seen as the bridge between instruction and learning.

“Moreover, the importance of reliable and valid measures for supporting teaching and learning is emphasized, and assessment is seen as the bridge between instruction and learning (William, 2014).”

• cites William, 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Norwegian ICT literacy framework states that its ambitions are to be included and operationalized in the subject-specific curricula, and the task is primarily given to the subject expert groups.

“It is stated in the Norwegian ICT literacy framework that the ambitions specified in the framework are to be included and operationalized in the subject-specific curricula, and the task is primarily given to the subject expert groups.”

May combine two claims.

The results exposed an imbalance between the theoretical frameworks and the operationalization of these and reveal that assessment is lagging behind.

“Hence, the results exposed an imbalance between the theoretical frameworks and the operationalization of these and reveal that assessment is lagging behind (Quellmalz, 2009).”

• cites Quellmalz, 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

These results emphasize the importance of taking the factor structure of the TEDDICS instrument into account.

“These results emphasize the importance of taking the factor structure of the TEDDICS instrument into account.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The attained level is also affected by the intended and implemented levels, because the intended curriculum is relevant for students' learning goals and what should be assessed.

“As shown in Figure 6, the attained level is also affected by the intended and implemented levels, because the intended curriculum is relevant for students' learning goals and what should be assessed.”

May combine two claims.

Both CIL and the 21st-century skills framework identify and delineate what is meant by ICT literacy, as well as group and define the underlying competence areas and the related competences.

“Both identify and delineate what is meant by ICT literacy, as well as group and define the underlying competence areas and the related competences.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

This enables the reporting of the separate dimensions, providing more detailed information that can be valuable for improved understanding of teaching and the alignment of the implemented and the intended curriculum.

“This enables the reporting of the separate dimensions, providing more detailed information that can be valuable for improved understanding of teaching and the alignment of the implemented and the intended curriculum.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The lack of a clear-cut ICT literacy curriculum may affect pre-service teachers and teacher training institutions.

“this lack of a clear-cut ICT literacy curriculum may also affect pre-service teachers and teacher training institutions”

• cites Tømte, Kårstein, & Olsen, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

The revised DIGCOMP framework functions adequately as a theoretical foundation and sufficiently delineates what ICT literacy should constitute from an educational viewpoint.

“the findings presented in the dissertation suggest that the revised DIGCOMP framework functions adequately as a theoretical foundation and sufficiently delineates what ICT literacy should constitute from an educational viewpoint.”

• AssumptionThe authors state this as an assumption: ICT literacy competences are to a great extent generic and not related to the technology itself.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Students' knowledge of ICT should be developed within the educational system.

“it has been emphasized that students' knowledge of ICT should be developed within the educational system”

• cites Griffin et al., 2012 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This may lessen some of the confusion caused by the use of different labels when addressing ICT in education.

“This may lessen some of the confusion caused by the use of different labels when addressing ICT in education.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both CIL and the 21st-century skills framework provide detailed descriptions of the content.

“Nevertheless, both CIL and the 21st-century skills framework provide detailed descriptions of the content.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The implemented level is dependent on the curriculum and/or educational standards (i.e., intended level), as they direct the arrangements of teaching and learning.

“The implemented level is dependent on the curriculum and/or educational standards (i.e., intended level), as they direct the arrangements of teaching and learning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The formal responsibility for instructing students to attain ICT literacy goals falls between different actors.

“it seems that the formal responsibility for instructing students to attain the ICT literacy goals falls between different actors”

• cites Tømte, Kårstein, & Olsen, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The theoretical frameworks play an important role because they are used to define and delimit test content and prior expectations about students' or teachers' conceptual understanding, which eventually guides item development.

“Besides, as described in the assessment development framework (section 2.3), the theoretical frameworks play an important role because these are used to define and delimit test content and prior expectations about students' or teachers' conceptual understanding, which eventually guides item development.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

What students learn at school (i.e., the implemented level) would most likely have an effect on their learning outcomes (e.g., achievement on the ICT literacy test).

“Moreover, what students learn at school (i.e., the implemented level) would most likely have an effect on their learning outcomes (e.g., achievement on the ICT literacy test).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The three levels of the curriculum model interact.

“Hence, the three levels interact (Goodlad et al., 1979).”

• cites Goodlad et al., 1979Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The third paper (LDN-ICT) investigated aspects of ICT literacy that have barely been operationalized in tests, although they are identified as critical for students to function in work life and the knowledge society.

“The third paper (LDN-ICT) investigated aspects of ICT literacy that have barely been operationalized in tests, although they are identified as critical for students to function in work life and the knowledge society (Binkley et al., 2012).”

• cites Binkley et al., 2012 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The four competence areas recommended in the Norwegian report The School of the Future are to a great extent in line with 21st-century skills frameworks.

“These competences are to great extent in line with the 21 st -century skills frameworks”

• cites Gallardo-Echenique et al., 2015Not checked — no full text of the source
• cites Griffin et al., 2012 sourceNot checked — no full text of the source
• cites Partnership for 21st Century Skills, 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

A detailed reporting at sub-construct level provides detailed information which could be used for designing classroom activities to increase students' learning of these competences, as well as by teachers for formative assessment purposes.

“However, a detailed reporting at sub-construct level provides detailed information which could be used for designing classroom activities to increase students' learning of these competences, as well as by teachers for formative assessment purposes.”

May combine two claims.

All three international frameworks are concerned with ICT literacy and address assessment as a vital aspect.

“However, all three are concerned with ICT literacy and address assessment as a vital aspect.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The three international frameworks (DIGCOMP, CIL, and 21st-century skills) underlying the three studies were developed within different, larger international enterprises, and they serve slightly different aims.

“The three international frameworks (i.e., DIGCOMP, CIL, and 21st-century skills) underlying the three studies were, as shown in Chapter 2, developed within different, larger international enterprises, and they serve slightly different aims.”

Classified as probably not a substantial claim.

The advantages of a multidimensional view were demonstrated above.

“And the advantages of a multidimensional view were demonstrated above.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

A reporting of the overall score on the ICT literacy tests provides essential information and could be useful to compare groups of students (e.g., age, gender, SES, ethnicity).

“A reporting of the overall score on the ICT literacy tests provides essential information and could be useful to compare groups of students (e.g., age, gender, SES, ethnicity).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The CIL framework is rather narrow compared to the revised DIGCOMP framework, while the 21st-century skills framework is fairly broad.

“While the CIL framework is rather narrow compared to the revised DIGCOMP framework, the 21st-century skills framework is fairly broad.”

May combine two claims.

The TEDDICS scale was developed within the large-scale international assessment ICILS, which mainly focuses on students' CIL as a subset of competences.

“This is not surprising, because it was developed within the large-scale international assessment ICILS, which mainly focuses on students' CIL as a subset of competences (Fraillon et al., 2014).”

• cites Fraillon et al., 2014 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

An alignment with the underlying framework and empirical investigations of the multidimensionality of it seem to be a fruitful approach.

“Therefore, an alignment with the underlying framework and empirical investigations of the multidimensionality of it seem to be a fruitful approach.”

Classified as probably not a substantial claim. May combine two claims.

These detailed results (based on the evidence for the four-dimensional Rasch model) indicate that the students need more training in some competence areas compared to others; this could be used to further inform instructional practice.

“These detailed results (based on the evidence for the four-dimensional Rasch model) indicate that the students need more training in some competence areas compared to others; this could be used to further inform instructional practice.”

Classified as probably not a substantial claim. May combine two claims.