EDM techniques for paper-based curricula

[Kevin Hall]

I am a teacher and curriculum author who plans to seek
research partners to evaluate the effectiveness of paper-based instructional
materials being designed in light of PSLC research. At the moment, the curriculum is still being
written and is not ready to be evaluated.
However, when it is ready, what EDM techniques could be used identify its
effective and ineffective elements in order to continually refine it? Because it is a paper-based curriculum, data
could most feasibly be taken during students’ tests and quizzes (for example,
if assessments are scantron-type, the data can easily be coded and logged to a
database).

The two most relevant articles I’ve read on using EDM to
identify effective and ineffective components of a curriculum are Feng,
Heffernan, & Beck [1] and Pavlik, Cen, & Koedinger [2]. I’m a classroom teacher, not an EDM
researcher, so please excuse any errors in the following analysis, but it seems
neither technique would work well on a paper-based curriculum. The reason is that both methods use learning
curve analysis. Learning curves can be
tracked for software-based curricula because the software logs every
interaction a student has with the program.
But if you’re just using paper-based quizzes/tests as data, you see only
a small subset of students’ attempts to solve problems: you see their attempts
on quizzes, but not during in-class discussions, homework problems, etc. You can probably record their attempts on at
most 10% of the problems they see. Can a
meaningful learning curve be generated and analyzed with such sporadic
data-taking? If not, how can authors of
paper-based curricula use EDM techniques to continually refine their materials?


Many school districts these
days require teachers to give scantron-type tests and quizzes. Commercial software tracks each student’s
skill level for each identified skill and reports progress back to teachers and
administrators. One such product is the
ExamView suite, which is widely used and which can be seen here: http://www.einstruction.com/pdf/brochures/K-12_ExamViewAS%20SS.pdf
. In my EDM readings, I have been
surprised not to find much research so far using the data collected from such
systems. I’m sure there are easy ways to
use the data to identify broad trends such as which schools/teachers/curricula
are on average more effective. But I’d
like to find a way to see inside a
curriculum, and to see which pieces of it work and which need to be
redesigned. In other words, my goal is
to “close the development loop” in authoring curricula.



Closing the development
loop is critical now because of a recent explosion in the number of open-source
curricula being written by volunteers and distributed over the web. The State of California is considering
shifting to such textbooks: http://www.scientificamerican.com/article.cfm?id=open-source-textbooks-mixed-bag-california. As all these new curricula are being developed,
how can the EDM community enable authors to test their materials’ effectiveness
without having to get major grants for controlled, experimental studies?



My own project, which is still in
its early stages, is available at the following URL if you’d like to see it:

http://www.curriki.org/xwiki/bin/view/Coll_Group_ImplementingAlgebra/ProportionsRatiosandPercents



Thank you to any EDM
community members who are willing to respond.
I always enjoy reading your research.



Kevin Hall



[1] Using
Learning Decomposition to Analyze Instructional Effectiveness in the ASSISTment
System

[2] Performance
Factors Analysis – A New Alternative to Knowledge Tracing







_________________________________________________________________
Ready for Fall shows? Use Bing to find helpful ratings and reviews on digital tv's.
http://www.bing.com/shopping/search?q=digital+tv's&form=MSHNCB&publ=WLHMTAG&crea=TEXT_MSHNCB_Vertical_Shopping_DigitalTVs_1x1

[Joseph E. Beck]

reason is that both methods use learning curve analysis. Learning curves
can be tracked for software-based curricula because the software logs every
interaction a student has with the program. But if you’re just using
paper-based quizzes/tests as data, you see only a small subset of students’
attempts to solve problems: you see their attempts on quizzes, but not
during in-class discussions, homework problems, etc. You can probably
record their attempts on at most 10% of the problems they see. Can a
meaningful learning curve be generated and analyzed with such sporadic
data-taking?

Actually, learning curve research done with intelligent tutors suffers
from a similar problem, especially if the tutor is used sporadically.
Imagine the following scenario:
September 16: use tutor and solve 3 problems on skill A (right, wrong, right)
November 15: use tutor and solve 3 more problems on skill A (wrong,
right, right)

[note: we do not need to assume that two months elapsed between uses
of the tutor, although that is one possibility. perhaps during the
intervening time the student worked on different skills on the tutor]

There were a couple months of "something" happening between the 3rd
and 4th attempts at solving problems with skill A. Odds are the
students saw examples in class, solved homework problems, etc. The
typically assumption is those anomalies even out over time. But I'm
not aware of anyone testing it, or proposing a good workaround.

If not, how can authors of paper-based curricula use EDM
techniques to continually refine their materials?

One thought would be to create fictional practice opportunities that
are unobserved. The EDMer has no idea how the student actually did on
those practice opportunities, but still counts them as opportunities.
A dynamic Bayesian research would simply add additional time slices
but have the observed node as unobserved. For the above data on skill
A, a traditional learning curve approach could have something like
Practice opportunity Correct?
1 yes
2 no
3 yes
23 (i.e. 4+"19") no
24 yes
25 yes

Where the "19" as an estimate of how much practice the student had in
the intervening months. The big question would be deciding how many
phantom practice opportunities to include. One solution would be
whatever number gives the smoothest learning curve (subject to some
bounds such as 0<=N<=1000), and insist that all members of a class
have the same value (or treat it as a mixed effects model).

Actually, estimating phantom practice opportunities based on learning
curves being lawful sounds like a reasonable and interesting approach.
Any grad students with some spare cycles out there? :-)

Many school districts these days require teachers to give scantron-type
tests and quizzes. Commercial software tracks each student’s skill level
for each identified skill and reports progress back to teachers and
administrators. One such product is the ExamView suite, which is widely
http://www.einstruction.com/pdf/brochures/K-12_ExamViewAS%20SS.pdf
. In my EDM readings, I have been surprised not to find much research so
far using the data collected from such systems.

A big reason is that EDM grew out of the ITS and AIED communities, so
the initial push was intelligent tutoring systems researchers. We're
picking up some people who use traditional data sources, but it's a
minority of the population. Also, the field is new enough that there
are plenty of low-hanging fruit, so why not work with a data source
that is plentiful?

I’m sure there are easy ways to use the data to identify broad trends such as which
schools/teachers/curricula are on average more effective.

Yes. Tread carefully though, as some teachers and schools don't want
ratings such as those available. On a prior project I came up with
some of those for internal consumption.

joe