research-article

Setting the Scope of Concept Inventories for Introductory Computing Subjects

Authors:

Geoffrey L. Herman,

Lisa Kaczmarczyk,

Michael C. Loui,

Craig ZillesAuthors Info & Claims

ACM Transactions on Computing Education (TOCE), Volume 10, Issue 2

Article No.: 5, Pages 1 - 29

https://doi.org/10.1145/1789934.1789935

Published: 01 June 2010 Publication History

Abstract

A concept inventory is a standardized assessment tool intended to evaluate a student’s understanding of the core concepts of a topic. In order to create a concept inventory it is necessary to accurately identify these core concepts. A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete mathematics, programming fundamentals, and logic design. The topic rankings can not only be used to guide the coverage of concept inventories, but can also be used by instructors to identify what topics merit special attention.

References

[1]

Allen, M. J. and Yen, W. M. 2002. Introduction to Measurement Theory. Long Grove, Ill.: Waveland Press.

[2]

Almstrum, V. L., Henderson, P. B., Harvey, V., Heeren, C., Marion, W., Riedesel, C., Soh, L.-K., and Tew, A. E. 2006. Concept inventories in computer science for the topic discrete mathematics. In Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE’06). 163--167.

Digital Library

[3]

Bransford, J. D., Brown, A. L., and Cocking, R. R. 1999. How People Learn: Brain, Mind, Experience, and School. National Academy Press.

[4]

Buck, J. R., Wage, K. E., Hjalmarson, M. A., and Nelson, J. K. 2007. Comparing student understanding of signals and systems using a concept inventory, a traditional exam, and interviews. In Proceedings of the 37th ASEE/IEEE Frontiers in Education Conference (ASEE’07). S1G1--S1G6.

[5]

Clark, R., Feldon, D., Merrienboer, J. V., Yates, K., and Early, S. 2008. Cognitive task analysis. In Handbook of Research on Educational Communications and Technology 3rd Ed., J. Spector, M. Merrill, J. van Merrienboer, and M. Driscoll, Eds. Lawrence Erlbaum Associates, Mahwah, NJ. 1801--1856.

[6]

Clayton, M. J. 1997. Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educ. Psychol. 17, 373--386.

[7]

Dalkey, N. and Helmer, O. 1963. An experimental application of the Delphi Method to the use of experts. Manage. Sci. 9, 458--467.

[8]

Evans, D. 2006. Personal communication.

[9]

Evans, D. et al. 2003. Progress on concept inventory assessment tools. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education (ASEE’03).

[10]

Glaser, B. and Strauss, A. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine, Chicago.

[11]

Goldman, K., Gross, P., Heeren, C., Herman, G., Kaczmarczyk, L., Loui, M. C., and Zilles, C. 2008. Identifying important and difficult concepts in introductory computing courses using a delphi process. In Proceedings of the 39th Annual ACM Technical Symposium on Computer Science Education (SIGCSE’08). 256--260.

Digital Library

[12]

Gray, G. L., Evans, D., Cornwell, P., Costanzo, F., and Self, B. 2003. Toward a nationwide dynamics concept inventory assessment test. In Proceedings of the American Society of Engineering Education Annual Conference (ASEE’03).

[13]

Hake, R. 1998. Interactive-engagement vs traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. Amer. J. Physics 66, 64--74.

[14]

Heffernan, N. and Koedinger, K. R. 1998. A developmental model for algebra symbolization: The results of a difficulty factors assessment. In Proceedings of the 20th Annual Conference of the Cognitive Science Society (CogSci’98). 484--489.

[15]

Herman, G. L., Kaczmarczyk, L., Loui, M. C., and Zilles, C. 2008. Proof by incomplete enumeration and other logical misconceptions. In Proceedings of the 4th International Computing Education Research Workshop (ICER’08). 59--70.

Digital Library

[16]

Herman, G. L., Loui, M. C., and Zilles, C. 2010. Creating the Digital Logic Concept Inventory. In Proceedings of the 41st Annual ACM Technical Symposium on Computer Science Education (SIGCSE’10). (In Press).

Digital Library

[17]

Herman, G. L., Zilles, C., and Loui, M. C. 2009. Work in progress: Students’ misconceptions about state in digital systems. In Proceedings of the 39th ASEE/IEEE Frontiers in Education Conference (ASEE’09).

Digital Library

[18]

Huettel, L. G., Forbes, J., Franzoni, L., Malkin, R., Nadeau, J., and Ybarra, G. 2007. Using tablet PCs to enhance engineering and computer science education. In The Impact of Tablet PCs and Pen-Based Technology on Education, J. C. Prey, R. H. Reed, and D. A. Berque, Eds. Purdue University Press. 59--66.

[19]

Jacobi, A., Martin, J., Mitchell, J., and Newell, T. 2003. A concept inventory for heat transfer. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education (ASEE’03).

[20]

Kaczmarczyk, L., Petrick, E., East, J. P., and Herman, G. L. 2010. Identifying student misconceptions of programming. In Proceedings of the 41st Annual ACM Technical Symposium on Computer Science Education (SIGCSE’10).

Digital Library

[21]

Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, A. A. R., Moström, J. E., Sanders, K., Seppälä, O., Simon, B., and Thomas, L. 2004. A multinational study of reading and tracing skills in novice programmers. In Proceedings of Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR’04). 119--150.

Digital Library

[22]

Marion, B. and Baldwin, D. 2007. SIGCSE commitee report: On the implementation of a discrete mathematics course. SIGCSE Bull. 39, 2, 109--126.

[23]

McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., and Wilusz, T. 2001. A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. In Proceedings of Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR’01). 125--180.

Digital Library

[24]

Mestre, J. P. 2005. Facts and myths about pedagogies of engagement in science learning. Peer Rev. 7, 2, 24--27.

[25]

Olds, B. M., Streveler, R. A., Miller, R. L., and Nelson, M. A. 2004. Preliminary results from the development of a concept inventory thermal and transport science. In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE’04).

[26]

Pill, J. 1971. The Delphi method: Substance, context, a critique and an annotated bibliography. Socio-Econ. Plan. Sci. 5, 1, 57--71.

[27]

Rowe, G. and Smaill, C. 2008. Development of an elecromagnetics course concept inventory. In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE’08).

[28]

Simon, B., Kohanfars, M., Lee, J., Tamayo, K., and Cutts, Q. 2010. Experience report: Peer instruction in introductory computing. In Proceedings of the 41st Annual ACM Technical Symposium on Computer Science Education (SIGCSE’10).

Digital Library

[29]

Soloway, E. and Spohrer, J. C. 1988. Studying the Novice Programmer. Lawrence Erlbaum Associates, Inc., Mahwah, NJ.

Digital Library

[30]

Stone, A., Allen, K., Rhodes, T. R., Murphy, T. J., Shehab, R. L., and Saha, C. 2003. The statistics concept inventory: A pilot study. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education (ASEE’03).

[31]

Streveler, R., Olds, B. M., Miller, R. L., and Nelson, M. A. 2003. Using a Delphi study to identify the most difficult concepts for studentsto master in thermal and transport science. In Proceddings of the American Society of Engineering Education, Annual Conference and Exposition (ASEE’03).

[32]

Tew, A. E., McCracken, W. M., and Guzdial, M. 2005. Impact of alternative introductory courses on programming concept understanding. In Proceedings of the 1st International Workshop on Computing Education Research (ICER’05). 25--35.

Digital Library

[33]

The Computer Society of the Institute for Electrical and Electronic Engineers and Association for Computing Machinery. 2001. Computing Curricula 2001, Computer science volume. http://www.sigcse.org/cc2001/index.html.

[34]

Zendler, A. and Spannagel, C. 2008. Empirical foundation of central concepts for computer science education. J. Educ. Res. Comput. 2, 6, 1--615.

Digital Library

Cited By

Lu KKrishnamurthi S(2024)Identifying and Correcting Programming Language Behavior MisconceptionsProceedings of the ACM on Programming Languages10.1145/36498238:OOPSLA1(334-361)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649823
Mazumder SPérez Quiñones MStephenson BStone JBattestilli LRebelsky SShoop L(2024)The Correctness of the Mental Model of Arrays After Instruction for CS1 StudentsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630943(806-811)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630943
Zhang HPerry ALee I(2024)Developing and Validating the Artificial Intelligence Literacy Concept Inventory: an Instrument to Assess Artificial Intelligence Literacy among Middle School StudentsInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00398-xOnline publication date: 5-May-2024
https://doi.org/10.1007/s40593-024-00398-x
Show More Cited By

Index Terms

Setting the Scope of Concept Inventories for Introductory Computing Subjects
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
2. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language types

Recommendations

Creating the digital logic concept inventory
SIGCSE '10: Proceedings of the 41st ACM technical symposium on Computer science education

A concept inventory (CI) is a standardized assessment tool that evaluates how well a student's conceptual framework matches the accepted conceptual framework of a discipline. In this paper, we present our process in creating and evaluating the alpha ...
Identifying important and difficult concepts in introductory computing courses using a delphi process
SIGCSE '08: Proceedings of the 39th SIGCSE technical symposium on Computer science education

A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing ...
Identifying important and difficult concepts in introductory computing courses using a delphi process
SIGCSE 08

A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computing Education

ACM Transactions on Computing Education Volume 10, Issue 2

June 2010

95 pages

EISSN:1946-6226

DOI:10.1145/1789934

Issue’s Table of Contents

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2010

Accepted: 01 March 2010

Revised: 01 February 2010

Received: 01 May 2009

Published in TOCE Volume 10, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

59
Total Citations
View Citations
1,123
Total Downloads

Downloads (Last 12 months)88
Downloads (Last 6 weeks)10

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lu KKrishnamurthi S(2024)Identifying and Correcting Programming Language Behavior MisconceptionsProceedings of the ACM on Programming Languages10.1145/36498238:OOPSLA1(334-361)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649823
Mazumder SPérez Quiñones MStephenson BStone JBattestilli LRebelsky SShoop L(2024)The Correctness of the Mental Model of Arrays After Instruction for CS1 StudentsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630943(806-811)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630943
Zhang HPerry ALee I(2024)Developing and Validating the Artificial Intelligence Literacy Concept Inventory: an Instrument to Assess Artificial Intelligence Literacy among Middle School StudentsInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00398-xOnline publication date: 5-May-2024
https://doi.org/10.1007/s40593-024-00398-x
Ali MGhosh SRao PDhegaskar RJawort SMedler AShi MDasgupta S(2023)Taking Stock of Concept Inventories in Computing Education: A Systematic Literature ReviewProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600120(397-415)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600120
Schmellenkamp MLatys AZeume TDoyle MStephenson BDorn BSoh LBattestilli L(2023)Discovering and Quantifying Misconceptions in Formal Methods Using Intelligent Tutoring SystemsProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569806(465-471)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569806
Mazumder SPérez-Quiñones M(2023)Incoming CS1 Students' Misconceptions on Arrays2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10342927(1-9)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10342927
Malmi LSheard JKinnunen PSimon Sinclair J(2022)Development and Use of Domain-specific Learning Theories, Models, and Instruments in Computing EducationACM Transactions on Computing Education10.1145/353022123:1(1-48)Online publication date: 29-Dec-2022
https://dl.acm.org/doi/10.1145/3530221
Getseva VKumar ABecker BQuille KLaakso MBarendsen ESimon (2022)An Empirical Analysis of Code-Tracing ConceptsProceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 110.1145/3502718.3524794(262-268)Online publication date: 7-Jul-2022
https://dl.acm.org/doi/10.1145/3502718.3524794
Sung SAlon LCho JKizilcec R(2022)How to Assess Student Learning in Information Science: Exploratory Evidence from Large College CoursesProceedings of the Association for Information Science and Technology10.1002/pra2.65959:1(500-504)Online publication date: 14-Oct-2022
https://doi.org/10.1002/pra2.659
Wiegand RBucci AKumar AAlbert JGaspar A(2021)Identifying Informatively Easy and Informatively Hard ConceptsACM Transactions on Computing Education10.1145/347796822:1(1-28)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3477968
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents