| Institution: |
Slovak University of Technology
|
| Technologies used: |
C#.NET
|
| Inputs: |
User (programmer) characteristics, User programming exercise activity log
|
| Outputs: |
Summative Assessment of Programmer’s Expertise
|
Addressed problem
It is difficult to evaluate programmer’s skills and abilities based on what activities he/she conducts. We present a novel method for evaluating programmer’s ability based on small programming exercises worked out in integrated development environment (IDE). We evaluated the method on 770 students and achieved accuracy of prediction within 0.57 to 1.02 grade levels.
Description
In our research we aim to create a novel method for automatic assessment of programmer’s skills and abilities based on his\hers work in an integrated development environment (IDE). Currently programmer’s skills and expertise are typically evaluated using static code analysis and programmer’s contributions to the code tree. We propose to evaluate programmer’s skills based on small programming exercises.
Using the data on exercises’ solution correctness we calibrate two-parametric IRT (Item Response Theory) logistic regression model, which models probability that a random programmer with ability theta will correctly solve i-th exercise. We evaluated the method on data from 770 students, and we achieved prediction accuracy of 0.57 to 1.02 grade levels to the real human assigned grades, while 72 to 90% of the grades were predicted within one grade level.
The IRT modelling provides us with a statistically sound way of aligning programmer abilities with programming exercises based on their difficulty. In order to apply this method in practice, we need a set of programming exercises of varying difficulties across the whole range of abilities.