Soon BrainPOP will be launching Assisted Grading for BrainPOP Science to make grading even easier and help teachers save time. Here are some answers to common questions about BrainPOP Assisted Grading to help you prepare!

IN THIS ARTICLE

About BrainPOP Assisted Grading
Safety and Security
Accuracy and Flexibility
Turning Assisted Grading on or off and Opting Out
The Future of Assisted Grading

About BrainPOP Assisted Grading

What is BrainPOP Assisted Grading?

Assisted Grading is BrainPOP’s AI-powered grading tool. It’s an automated feature designed to save teachers time and make grading easier! Assisted Grading analyzes students’ Claim-Evidence Reasoning (CER) responses using the BrainPOP Science rubric, and provides suggested scores and feedback. Teachers can choose to apply or modify any of these suggestions.

How does BrainPOP Assisted Grading work?

BrainPOP Assisted Grading uses a large language model (LLM) to analyze the content of each student's response. Using large amounts of data, the LLM has been trained to understand human-written information. Our algorithm compares a student’s response to the CER rubric and generates a score based on criteria such as scientific accuracy, organization, coherence, and language proficiency.

How the model works:

1. The following information is entered (input) into the model: 
   • Investigation title, Guiding Question, and context based on the Investigation’s learning design
   • Students’ CER responses
   • The CER scoring rubric
   • CER examples scored by our team of experts to fine-tune the model
     
     
2. The LLM assesses students’ work. The model has been tuned to this task by using human-scored CER examples.
   
   
3. The model then outputs:
   • Scores for each CER section and its subsections
   • Written rationales, or explanations, for each score for teachers to review

Teachers can accept or override any of the Assisted Grading score suggestions. Students will only see scores that have been teacher-approved. If there are any issues with the suggested scores, we encourage teachers to send us feedback so that we can improve the algorithm.

Which large language model (LLM) does BrainPOP use?

Because of its privacy and safety features, BrainPOP uses a custom model using Google’s Vertex AI LLM. Input is not shared outside of the customer model. Personally identifiable information is never used to train the AI models. It is also fast, flexible, and rigorously evaluated for potential harms and biases. 

Does BrainPOP Assisted Grading work in languages other than English?

As of now, Assisted Grading is only offered in English. We may expand to other languages in the future.

Safety and Security

How are we combating algorithmic bias?

Our model was fine-tuned using a set of responses from a diverse student population. The scored CERs used to adapt the model were hand-graded by our team of Learning Design experts, including doctors of science education and former science teachers. 

We used various methods to ensure accuracy in scores, including a scoring calibration training and scoring spot checks. We also assessed inter-rater reliability using the quadratic weighted kappa statistic to measure agreement between hand-scored CERs and the automated scoring model. 

How is student data secured and student data privacy implemented when using BrainPOP Assisted Grading?

We do not send any personally identifiable student information to the LLM. We adhere to strict data protection standards and comply with relevant privacy regulations. Personally identifiable student data is encrypted, securely stored, and accessible only to authorized BrainPOP personnel. All student data stays within our “walled garden,” which means it is not fed into any open-source models on the internet.

Is student data going to be used to train AI models?

Student responses will not be used to train any models outside of BrainPOP. De-identified or anonymous student information may be used to train BrainPOP’s AI model for the purposes of improving the BrainPOP Assisted Grading feature. 

Are students going to interact with BrainPOP Assisted Grading in any way?

No. The teacher is the only one who will see the AI-suggested scores. And of course, teachers can modify the score at any time.

For more information on our privacy and security practices, including FERPA and COPPA compliance, please visit our privacy policy.

Accuracy and Flexibility

Will BrainPOP Assisted Grading completely replace teacher grading?

Nope. Assisted Grading is meant to enhance and expedite the grading process, but teacher judgment and expertise are still crucial. While AI-suggested grading is designed to help teachers save time, it doesn’t completely replace human grading. Assisted Grading provides scoring suggestions that teachers can choose to accept or override.

How accurate is BrainPOP Assisted Grading?

Assisted Grading is smart, but not perfect. Based on our research and testing, we’re confident Assisted Grading provides consistent and reliable feedback. However, it may not capture certain nuances that require a teacher’s judgment. We encourage teachers to send us feedback on scores they modify so we can continue to improve the model.

Turning Assisted Grading on or off and Opting Out

How do I turn BrainPOP Assisted Grading on or off?

Teachers can turn Assisted Grading on or off by going to Site Settings and locating the Assisted Grading panel. Move the toggle switch on or off. This will control the Assisted Grading tool for all CERs. Please note that the de-identified student responses will continue to be processed by the AI model in the backend. 

If a school or district does not want the student data in the responses to be processed by the tool, the administrator on the account can request to opt-out by emailing legal@brainpop.com. Admin opt outs will prevent all teachers from using Assisted Grading, as well as block the de-identified student data from the responses from being sent to the AI model.

The Future of Assisted Grading

Am I able to bulk accept AI suggestions?

While this feature is in beta, teachers need to review and accept the AI suggestions to help validate that everything is working as intended. We hope to be able to provide bulk accepting capabilities soon.

Can students see the suggested scores before submission?

Not at this time. But in the future, we would like to leverage this technology to help students improve their CER writing skills before submission.

Are students going to see the rationales for the AI-generated grades?

Not yet. For now, students can see teacher feedback and the description of points under the points section in the rubric table. Teachers can copy and paste the AI rationale to provide to students as feedback if they’d like.

Does BrainPOP Science plan to extend BrainPOP Assisted Grading to other parts of the experience?

Right now, we’re experimenting with Assisted Grading for CER and receiving feedback from teachers about its use. In the future, we hope to expand this feature to more areas to provide more grading help for teachers.