Quickly Producing "Isomorphic" Exercises: Quantifying the Impact of Programming Question Permutations
Abstract
Small, auto-gradable programming exercises provide a useful tool with which to assess students' programming skills in introductory computer science. To reduce the time needed to produce programming exercises of similar difficulty, previous research has applied a permutation strategy to existing questions. Prior work has left several open questions: is prior exposure to a question typically indicative of higher student performance? Are observed changes in difficulty due to the specific surface feature permutations applied? How is student performance impacted by the first version of a question to which they may be exposed?
In this work, we pursue this permutation strategy in multiple semesters of an introductory Python course to investigate these open questions. We use linear regression models to tease out the impacts of different surface feature changes and usage conditions. Our analysis finds similar tendencies as in prior work: question versions available in study materials tend to have 5 -- 11 percentage point higher scores than novel permutations and more ''substantial'' surface feature changes tend to produce harder questions. Our results suggest this last finding is sensitive to how evenly permutations are applied across existing questions, as the precise impact of individual permutations changes between semesters.
References
[1]
Kirsti M Ala-Mutka. 2005. A Survey of Automated Assessment Approaches for Programming Assignments. Computer Science Education, Vol. 15, 2 (2005), 83--102. https://doi.org/10.1080/08993400500150747
[2]
Betsy Bizot and Stu Zweben. 2019. Generation CS, Three Years Later. Technical Report. Computing Research Association. https://cra.org/generation-cs-three-years-later/
[3]
Paul D. Bliese, David Chan, and Robert E. Ployhart. 2007. Multilevel Methods: Future Directions in Measurement, Longitudinal Analyses, and Nonnormal Outcomes. Organizational Research Methods, Vol. 10, 4 (2007), 551--563. https://doi.org/10.1177/1094428107301102
[4]
Dennis Bouvier, Ellie Lovellette, John Matta, Bedour Alshaigy, Brett A. Becker, Michelle Craig, Jana Jackova, Robert McCartney, Kate Sanders, and Mark Zarb. 2016. Novice Programmers and the Problem Description Effect. In Proceedings of the 2016 ITiCSE Working Group Reports (ITiCSE '16). Association for Computing Machinery, New York, NY, USA, 103--118. https://doi.org/10.1145/3024906.3024912
[5]
Liia Butler, Geoffrey Challen, and Tao Xie. 2020. Data-Driven Investigation into Variants of Code Writing Questions. In 2020 IEEE 32nd Conference on Software Engineering Education and Training (CSEE&T). 1--10. https://doi.org/10.1109/CSEET49119.2020.9206195 ISSN: 2377--570X.
[6]
Tracy Camp, W. Richards Adrion, Betsy Bizot, Susan Davidson, Mary Hall, Susanne Hambrusch, Ellen Walker, and Stuart Zweben. 2017. Generation CS: The Mixed News on Diversity and the Enrollment Surge. ACM Inroads, Vol. 8, 3 (July 2017), 36--42. https://doi.org/10.1145/3103175
[7]
Michelene T. H. Chi, Paul J. Feltovich, and Robert Glaser. 1981. Categorization and Representation of Physics Problems by Experts and Novices*. Cognitive Science, Vol. 5, 2 (1981), 121--152. https://doi.org/10.1207/s15516709cog0502_2
[8]
Michael J. Clancy and Marcia C. Linn. 1999. Patterns and Pedagogy. In The Proceedings of the Thirtieth SIGCSE Technical Symposium on Computer Science Education (New Orleans, Louisiana, USA) (SIGCSE '99). ACM, New York, NY, USA, 37--42. https://doi.org/10.1145/299649.299673
[9]
Computing Research Association. 2017. Generation CS: Computer Science Undergraduate Enrollments Surge Since 2006. Technical Report. https://cra.org/data/generation-cs/
[10]
Michelle Craig, Jacqueline Smith, and Andrew Petersen. 2017. Familiar contexts and the difficulty of programming problems. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research (Koli Calling '17). Association for Computing Machinery, New York, NY, USA, 123--127. https://doi.org/10.1145/3141880.3141898
[11]
Raquel M Crespo, Jad Najjar, Michael Derntl, Derick Leony, Susanne Neumann, Petra Oberhuemer, Michael Totschnig, Bernd Simon, Israel Gutierrez, and Carlos Delgado Kloos. 2010. Aligning assessment with learning outcomes in outcome-based education. In IEEE EDUCON 2010 Conference. IEEE, 1239--1246.
[12]
Janet E Davidson and Robert J Sternberg. 2003. The psychology of problem solving. Cambridge university press.
[13]
Jens Dolin, Paul Black, Wynne Harlen, and Andrée Tiberghien. 2018. Exploring relations between formative and summative assessment. In Transforming assessment. Springer, 53--80.
[14]
Stephen H. Edwards and Manuel A. Perez-Quinones. 2008. Web-CAT: Automatically Grading Programming Assignments. In Proceedings of the 13th Annual Conference on Innovation and Technology in Computer Science Education (Madrid, Spain) (ITiCSE '08). Association for Computing Machinery, New York, NY, USA, 328. https://doi.org/10.1145/1384271.1384371
[15]
James Finnie-Ansley, Paul Denny, and Andrew Luxton-Reilly. 2021. A Semblance of Similarity: Student Categorisation of Simple Algorithmic Problem Statements. In Proceedings of the 17th ACM Conference on International Computing Education Research (ICER 2021). Association for Computing Machinery, New York, NY, USA, 198--212. https://doi.org/10.1145/3446871.3469745
[16]
Daniel T. Fokum, Daniel N. Coore, Eyton Ferguson, Gunjan Mansingh, and Carl Beckford. 2019. Student Performance in Computing Courses in the Face of Growing Enrollments. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE '19). Association for Computing Machinery, New York, NY, USA, 43--48. https://doi.org/10.1145/3287324.3287354
[17]
Max Fowler and Craig Zilles. 2021. Superficial Code-guise: Investigating the Impact of Surface Feature Changes on Students' Programming Question Scores. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE '21). Association for Computing Machinery, New York, NY, USA, 3--9. https://doi.org/10.1145/3408877.3432413
[18]
John R Hayes. 1977. Psychological differences among problem isomorphs. Cognitive theory, Vol. 2 (1977), 21--41.
[19]
Eder Hernandez, Esmeralda Campos, Pablo Barniol, and Genaro Zavala. 2019. The effect of similar surface features on studentstextquoteright understanding of the interaction of charges with electric and magnetic fields. In Physics Education Research Conference, PERC 2019 (Physics Education Research Conference Proceedings), Ying Cao, Steven Wolf, and Michael Bennett (Eds.). American Association of Physics Teachers, United States, 220--225. https://doi.org/10.1119/perc.2019.pr.Hernandez
[20]
Matthew Inglis and Lara Alcock. 2012. Expert and Novice Approaches to Reading Mathematical Proofs. Journal for Research in Mathematics Education, Vol. 43, 4 (2012), 358--390. https://doi.org/10.5951/jresematheduc.43.4.0358
[21]
Eric Kjolsing and Lelli Van Den Einde. 2016. Peer Instruction: Using Isomorphic Questions to Document Learning Gains in a Small Statics Class. Journal of Professional Issues in Engineering Education and Practice, Vol. 142, 4 (2016), 04016005.
[22]
Juho Leinonen, Paul Denny, and Jacqueline Whalley. 2021. Exploring the Effects of Contextualized Problem Descriptions on Problem Solving. In Australasian Computing Education Conference (ACE '21). Association for Computing Machinery, New York, NY, USA, 30--39. https://doi.org/10.1145/3441636.3442302
[23]
Filip Lievens and Paul R. Sackett. 2007. Situational judgment tests in high-stakes settings: Issues and strategies with generating alternate forms. Journal of Applied Psychology, Vol. 92 (2007), 1043--1055. https://doi.org/10.1037/0021--9010.92.4.1043
[24]
Russell Millar and Sathiamoorthy Manoharan. 2021. Repeat individualized assessment using isomorphic questions: a novel approach to increase peer discussion and learning. International Journal of Educational Technology in Higher Education, Vol. 18, 1 (2021), 1--15.
[25]
Nona Muldoon and Chrisann Lee. 2007. Formative and summative assessment and the notion of constructive alignment. Enhancing teaching and learning through assessment (2007), 98--108.
[26]
Ross H. Nehm and Minsu Ha. 2011. Item feature effects in evolution assessment. Journal of Research in Science Teaching, Vol. 48, 3 (2011), 237--256. https://doi.org/10.1002/tea.20400
[27]
Ross H Nehm and Judith Ridgway. 2011. What do experts and novices "see" in evolutionary problems? Evolution: Education and Outreach, Vol. 4, 4 (2011), 666--679.
[28]
Allen Newell and Herbert A. Simon. 1972. Human problem solving. Prentice-Hall, Oxford, England. xiv, 920--xiv, 920 pages.
[29]
Peterson K Ozili. 2022. The acceptable R-square in empirical modelling for social science research. Available at SSRN 4128165 (2022).
[30]
Miranda C. Parker, Leiny Garcia, Yvonne S. Kao, Diana Franklin, Susan Krause, and Mark Warschauer. 2022. A Pair of ACES: An Analysis of Isomorphic Questions on an Elementary Computing Assessment. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22). Association for Computing Machinery, New York, NY, USA, 2--14. https://doi.org/10.1145/3501385.3543979
[31]
Miranda C. Parker, Mark Guzdial, and Shelly Engleman. 2016. Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment. In Proceedings of the 2016 ACM Conference on International Computing Education Research (Melbourne, VIC, Australia) (ICER '16). Association for Computing Machinery, New York, NY, USA, 93--101. https://doi.org/10.1145/2960310.2960316
[32]
Miranda C. Parker, Yvonne S. Kao, Dana Saito-Stehberger, Diana Franklin, Susan Krause, Debra Richardson, and Mark Warschauer. 2021. Development and Preliminary Validation of the Assessment of Computing for Elementary Students (ACES). In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE '21). Association for Computing Machinery, New York, NY, USA, 10--16. https://doi.org/10.1145/3408877.3432376
[33]
Jolynn Pek and David B. Flora. 2018. Reporting effect sizes in original psychological research: A discussion and tutorial. Psychological Methods, Vol. 23 (2018), 208--225. https://doi.org/10.1037/met0000126
[34]
Nancy Pennington. 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive psychology, Vol. 19, 3 (1987), 295--341.
[35]
Leo Porter, Cynthia Bailey Lee, Beth Simon, and Daniel Zingaro. 2011. Peer Instruction: Do Students Really Learn from Peer Discussion in Computing?. In Proceedings of the Seventh International Workshop on Computing Education Research (Providence, Rhode Island, USA) (ICER '11). Association for Computing Machinery, New York, NY, USA, 45--52. https://doi.org/10.1145/2016911.2016923
[36]
Frederick Reif. 2008. Applying Cognitive Science to Education: Thinking and Learning in Scientific and Other Complex Domains. MIT Press.
[37]
Skipper Seabold and Josef Perktold. 2010. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference.
[38]
Nicole A Shea, Ravit Golan Duncan, and Celeste Stephenson. 2015. A tri-part model for genetics literacy: Exploring undergraduate student reasoning about authentic genetics dilemmas. Research in Science Education, Vol. 45, 4 (2015), 485--507.
[39]
Rob Wass and Clinton Golding. 2014. Sharpening a tool for teaching: the zone of proximal development. Teaching in Higher Education, Vol. 19, 6 (2014), 671--684. https://doi.org/10.1080/13562517.2014.901958
[40]
Matthew West, Geoffrey L. Herman, and Craig Zilles. 2015. PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine Learning. 26.1238.1--26.1238.14. https://peer.asee.org/prairielearn-mastery-based-online-problem-solving-with-adaptive-scoring-and-recommendations-driven-by-machine-learning
[41]
Michele Weston, Kevin C. Haudek, Luanna Prevost, Mark Urban-Lurain, and John Merrill. 2015. Examining the Impact of Question Surface Features on Students' Answers to Constructed-Response Questions on Photosynthesis. CBE Life Sciences Education, Vol. 14, 2 (June 2015). https://doi.org/10.1187/cbe.14-07-0110
[42]
Aaron S. Yarlas and Vladimir M. Sloutsky. 2000. Problem Representation in Experts and Novices: Part 1. Differences in the Content Of Representation. In In. Erlbaum, 475--480.
[43]
Daniel Zingaro and Leo Porter. 2015. Tracking Student Learning from Class to Exam Using Isomorphic Questions. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (Kansas City, Missouri, USA) (SIGCSE '15). Association for Computing Machinery, New York, NY, USA, 356--361. https://doi.org/10.1145/2676723.2677239
Index Terms
- Quickly Producing "Isomorphic" Exercises: Quantifying the Impact of Programming Question Permutations
Recommendations
Superficial Code-guise: Investigating the Impact of Surface Feature Changes on Students' Programming Question Scores
SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science EducationAssessing student performance on programming questions is important for introductory computer science courses, both for student learning and for ensuring students demonstrate competence. Part of being a competent programmer includes the ability to ...
Evaluating Automatically Generated Contextualised Programming Exercises
SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1Introductory programming courses often require students to solve many small programming exercises as part of their learning. Researchers have previously suggested that the context used in the problem description for these exercises is likely to impact ...
Quantifying the Effects of Prior Knowledge in Entry-Level Programming Courses
CompEd '19: Proceedings of the ACM Conference on Global Computing EducationComputer literacy and programming are being taught increasingly at the K-12 level with more students than ever matriculating in college with prior programming experience. Accurately assessing student programming skills acquired in high school can inform ...
Comments
Information & Contributors
Information
Published In
July 2024
776 pages
ISBN:9798400706004
DOI:10.1145/3649217
- General Chairs:
- Mattia Monga,
- Violetta Lonati,
- Erik Barendsen,
- Program Chairs:
- Judithe Sheard,
- James Paterson
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ITiCSE 2024
Sponsor:
Acceptance Rates
Overall Acceptance Rate 552 of 1,613 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 54Total Downloads
- Downloads (Last 12 months)54
- Downloads (Last 6 weeks)18
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in