Do adaptive assessments disadvantage students?
There’s a lot of talk lately about the dismal reading and maths scores of American children. I’m sure we’ve all heard them. I’m also sure that we’ve heard that they’re trending downward. There’s panic at school boards. There’s panic at the state house. There’s panic in Congress, who is looking at increasing the budget of the Dept. of Education to “solve” this problem.
But do we actually know what the problem is?
If we don’t know what the actual problem is, how do we hope to “solve” it?
I say this as our summer school session draws to a close here. I’m teaching summer school and have those who are normally in a special day program (students whose IEPs call for them to be in segregated settings during the school year) in the Extended School Year programme.
At the beginning of the session, we took an assessment to see where the kids were in their academic journey. This was done on one of the District’s preferred apps. I received the results. The app said that my 8th and 9th graders were working between the 3rd grade and kindergarten level. I didn’t believe it.
Over the next few days, I gave them some worksheets that tested them on things the app said they didn’t know or couldn’t do. They got through them at their own pace … but they got the vast majority of the questions correct.
My worksheets don’t use deceptive wording. My worksheets are designed to test a specific skill. There are no trick questions, no attractive wrong answers, and no penalties for taking your time.
So what happened with the app?
The apps, in a general sense, aren’t testing skills as skills. They’re testing skills vs. time. This would disfavour not only my students, but students with learning or language disabilities.
Plus, as a student struggles with the wording of a question, or whose ADHD has distracted them for a time, the program assumes that they don’t know the answer. Regardless of the answer given, the system “adapts” and assigns a lower grade level question to the student. When this is repeated throughout the test, the student will begin at grade level, but end well below their grade level. Indeed, as I walked around the room as they were testing, most were finishing the assessment answering kindergarten-level questions.
Here’s what’s going on with these “adaptive assessments.”
Adaptive assessment involves dynamically adjusting the difficulty level of questions based on the test taker's performance. The aim, they say, is to provide a more tailored and personalized experience by presenting questions that are appropriately challenging for the individual.
In adaptive assessment, the initial set of questions might be of moderate difficulty. Based on the test taker's responses, including the time it takes to respond, the system analyzes their performance and determines the appropriate difficulty level for subsequent questions. If the test taker answers a question correctly, the system may present a more challenging question next. Conversely, if the answer is incorrect, or the student takes “too much time” to answer, the system will provide an easier question. This process continues throughout the assessment, with the system continually adapting the difficulty level to find an appropriate balance for the test taker.
Adaptive assessment are employed in various educational or assessment settings, such as online learning platforms, certification exams, or standardized tests, to “optimize” the assessment experience. The BigEd companies say that these assessments provide more accurate measures of knowledge or skills. But, practical experience here in Los Angeles says otherwise.
Which apps / companies use adaptive assessments?
Several educational apps for K-12 are known for their adaptive leveling. Here are a few examples:
1. Khan Academy: Khan Academy offers adaptive practice in subjects such as math, science, history, and more. The platform adjusts the difficulty of questions based on the student's performance, including time spent per response, and provides personalized recommendations.
2. IXL Learning: IXL Learning provides adaptive learning programs for various subjects, including math, language arts, science, and social studies. It adapts to the student's skill level and time spent per response, and adjusts the difficulty of questions accordingly.
3. DreamBox Learning: DreamBox Learning focuses on adaptive math learning for students in kindergarten through eighth grade. It provides “personalized lessons” and adjusts the difficulty level based on the accuracy and speed of the student's responses.
4. ScootPad: ScootPad is an adaptive learning platform for K-8 students, covering subjects like math, reading, and writing. It tailors the curriculum to each student's abilities and adjusts the difficulty of questions accordingly.
5. Lexia Learning: Lexia Learning specializes in adaptive reading programs for students in pre-K through grade 12. The app assesses students' skills and speed and provides personalized lessons and activities to improve reading proficiency.
These are just a few examples, and there are many other educational apps and platforms that employ adaptive leveling techniques to provide personalized learning experiences for K-12 students.
Ask the right questions.
Most adaptive platforms in the West do factor in student response time to questions as part of their adaptive algorithms. They say that response time provides valuable insights into a student's understanding and level of proficiency.
Here's how response time is often considered:
1. Difficulty Adjustment: Adaptive platforms use response time as an indicator of question difficulty. If a student answers a question quickly and correctly, they say it suggests a higher level of mastery, and the platform will present a more challenging question next. Conversely, if the student takes longer to answer or answers incorrectly, it will present a slightly easier question.
2. Skill Assessment: Response time can also help assess a student's fluency and automaticity in a particular skill. For example, in math, quick response times on basic arithmetic questions might indicate a strong foundation in that skill area. But, are we testing speed or accuracy? Which do we, as a society, value? Would you rather the engineer be fast, or accurate?
3. Time Limits: Again, most adaptive platforms impose time limits on questions. They say this is to encourage students to think and respond quickly. The platform will adjust the difficulty of subsequent questions based on how well the student performs within the given time constraints. If the student fails to respond within the time limit, the student will be penalized with a lower level of subsequent questions … regardless if the student answered the question correctly.
By considering response time, adaptive platforms favour a student's speed, adapt the difficulty of questions accordingly, and provide a learning experience that favours a specific type of student over others.
Challenges
Response time adjustments in adaptive platforms are challenging for students with language difficulties and learning disorders. Students who struggle with language processing or have limited proficiency in the language used in the assessment may take longer to understand the question, formulate a response, or navigate the interface. As a result, they will face difficulties in completing questions within the time limits imposed by the platform.
In such cases, it is important for adaptive platforms to consider the specific needs of students with language difficulties. But I, the teacher, do not have the ability to inform the platform of these unique needs. Neither do I have the ability to specify a students accommodations. If, for example, my student gets simplified test instructions or having the question read aloud by an adult, these will usually take more time that the app permits, resulting in a lowering of the question difficulty moving through the test.
Here are a few considerations if you’re in the market for such adaptive assessments:
Flexible Time Limits: Can the adaptive platform provide flexibility in time limits or eliminate time constraints altogether for students who require additional time due to language difficulties? This accommodation allows students to focus on understanding the question and formulating their responses without the added pressure of time restrictions.
Different Assessment Formats: Can platforms offer alternative assessment formats that cater to different learning styles or language abilities? For example, providing multiple-choice questions or interactive visuals alongside text-based questions can help students with language difficulties demonstrate their knowledge and skills in a more accessible way.
Individualized Adaptation: Can adaptive platforms adapt not only based on response time but also other indicators of proficiency, such as the accuracy of responses or a student's progress over time? By considering a holistic view of student performance, the platform can provide appropriate challenges and support tailored to each student's needs.
Language Support: Can adaptive platforms offer language support features such as translations, simplified language options, or audio narration to assist students with language difficulties in understanding the questions and instructions more effectively?
It is crucial for adaptive platforms to take into account the diverse needs of students, including those with language difficulties, and provide accommodations and supports that promote equitable learning experiences. It’s equally crucial that when using such platforms to assess students in relation to their peers (local / state / national / international), that the test and testing environment be fair.
So why work in this way?
It seems to me that these “testing companies” are creating a problem where none really exists. Is it a surprise, for example, that IXL offers monthly / yearly subscriptions to help bring students up to speed on their deficiencies? The term commonly used to describe the act of creating a problem in order to sell the solution is “manufactured or engineered demand.” It refers to the deliberate manipulation of consumer demand by creating a perceived need for a particular product or service. This tactic often involves exaggerating or inventing a problem and offering a solution that may not be necessary or effective. It is a controversial practice that can be considered unethical or manipulative, as it takes advantage of consumers’ vulnerabilities or fears for commercial gain.
“Capitalist Realism: Is There No Alternative?” is a 2009 book by Mark Fisher that examines the pervasive influence of neoliberal capitalism on contemporary society. While not specifically focused on “engineered demand,” the book explores how capitalism shapes and restricts our perceptions of what is possible and desirable.
Fisher argues that capitalist realism creates a sense of inevitability, making it difficult for people to imagine alternatives. It perpetuates the notion that there is no alternative to capitalism, leading to a resignation and acceptance of the status quo. Within this context, “engineered demand” can be seen as a symptom of capitalist realism, as it reinforces the idea that commercial solutions are the only viable ones, limiting our capacity to critically evaluate the problems being presented and seek alternative approaches - like the testing software isn’t fit for purpose and we don’t actually need these expensive “remedies.”
Fisher's work invites readers to question the ideological constraints imposed by capitalist realism and consider the possibilities for social change. By challenging the notion that there are no alternatives, it encourages a critical examination of the manufactured needs and desires created by the capitalist system, including the tactics of “engineered demand.”
Do adaptive assessments disadvantage students?
Consider my District’s enrollment. Of the 538,295 students, 114,523 of them are English language learners, and 84,722 of them have IEPs. There’s a lot of overlap between the ELL and IEP population. There is also a significant, but unreported number of students who reclassify out of the ELL program who still need supports. These numbers are can be teased out of or inferred from the various reports on the state’s dashboard. You’ll find that my school is on the High band of schools where students make progress towards their language goals a majority of the time.
But either / both represent a significant number of students. What if the stats are off by 20% - meaning what if the ~20% of the student body who likely does not interact well with the assessment software test well below grade level? How will that skew the data?
Conclusion
As we get ready to close out the summer holiday, teachers and administrators are already preparing for the first weeks of school. One of the first tasks is to administer the Beginning of Year assessments with the students. How do you think that will go? What will you think about the results?