Lessons Learned From Developing High-volume, High-stakes, High-quality Items

Writing assessments is a challenging task. As educators, we want to develop assessment items that engage our learners and challenge them to think critically. Assessment items can take many different forms, from the classic multiple-choice item to a performance-based task that asks the learner to apply skills to a realistic scenario. According to the National Center for Education Statistics, a quality assessment item “allows valid inferences about student knowledge and skills.” In other words, we need to be able to confidently say that our assessments test the right material. A quality assessment item needs to:

• align to learning objectives in the lesson
• challenge the learner to use higher-level thinking skills
• engage and motivate the learner
• measure what it is supposed to (content validity)

The stakes for creating quality assessment items become higher as the quantity increases. My company, A Pass Educational Group, LLC, has rapidly grown from a few freelancers and myself in 2010 to the over 1,500 contractors and full-time employees we have today. Over the years, we have developed thousands of assessment items for different clients. This growth has challenged me to scale the assessment-writing process without compromising quality. I have learned a few lessons in overseeing the many projects that my company has completed, and my determination to develop quality items has challenged me to design a better process for developing assessments.

Define roles

I used to receive emails from my colleagues expressing confusion as to what they should be doing. Maybe they thought they were only writing assessment items and weren’t also responsible for reviewing their colleague’s work. Stopping to clarify roles not only delays the project; it can lead to frustrated employees. With a larger team working on assessment items, it is even more important to define expectations for each team member. I have found that including a detailed outline of roles and responsibilities for each team member at the beginning of a project prevents this confusion. Along with that outline of roles and responsibilities, we include a breakdown of what the workflow will look like and where each key player will engage in the process. The result of taking this step before the project begins is that team members feel more confident as to how to meet expectations and take ownership of their roles.

Set expectations

This has been a big lesson for me in working with clients on assessments. You always have to ask yourself, “What are the client’s expectations?” I have found that each client has a specific format and style for the way they prefer to assess their students. Sometimes the client’s example assessment items test lower-level thinking skills, like identify and define, without testing higher-level skills such as analyzing and creating original responses. We then work with the client by providing examples of our work that reach those higher-level thinking skills in order to provide the highest-quality product for the client. Ultimately, the client determines what they want and need from us, and we organize our efforts to meet those expectations. It is critical that we achieve a mutual understanding of those needs and expectations before our work begins. Setting and agreeing upon the expectations at the launch of the project helps prevent any confusion later.

Check your Alignment

It can be incredibly frustrating to write an assessment item that you feel will knock it out of the park, only to realize it does not align with the content. This mistake is easier to make as the volume of projects you work on increases. One way to double-check is to include documentation of which objective your assessment covers. You can write an agreed-upon key prior to development of the assessment, with helpful abbreviations for each objective. As you write your assessment, include documentation of what objective is covered. This will help you evaluate the effectiveness of the assessment item, and your client will rest assured that the assessment items test your objectives. This communication is key throughout the development process. It begins with communicating to the writer the objective to which the item should align, continues with the editor checking for alignment, and extends to communicating to the client what the intended objective is for each item based upon the assignments provided in the client blueprint.

Document, Track, Collaborate

It can seem chaotic at times to manage hundreds of assessment items for multiple projects. I have found that being able to track all changes makes a big difference. Google Docs has allowed my teams to not only collaborate with one another but also easily view the history of all changes over the course of a project. As a project manager, you can assign your team members specific tasks to complete within the document and track progress. You can also easily share any document with a client and highlight any areas that require further clarification or collaboration.

Evaluation

A well-documented and planned evaluation cycle is critical to ensuring quality assessment items are developed at the highest level of quality. With a small project and a manageable amount of assessment items, this task is a lot easier. A bigger project with higher stakes requires more planning. When you create a project timeline, make sure you allocate enough time for evaluation. In the evaluation process, there need to be multiple eyes looking at the items for quality assurance. It may be tempting in the interest of time to have only an editor review a writer’s work, but it is important that there be multiple levels of review. A content editor will check for key elements like standard alignment, plausible answer choices, and clear rationales. Meanwhile, a project manager has had more communication with the client and will look for different requirements than the editor. A copy editor may notice that there are grammatical or formatting errors. Each key player in the process is critical to the quality and success of the project.

Final Thoughts

I have learned a lot over the years as my project load has exponentially increased. The difficulties I faced as I scaled the way I develop assessments has taught me important lessons. Careful documentation, setting expectations, and striving for quality have been major driving factors in how I have approached developing assessments. No matter how many assessment items you are responsible for developing, a careful approach to your development process will ensure you do not lose sight of quality.

Enjoy this checklist developed by A Pass that enables those developing performance based items, and those reviewing them, to ensure that students encounter assessments that will both evaluate their knowledge and skills as well as advance their learning.

Sources

https://www2.ed.gov/policy/elsec/leg/essa/essanceshighqualityassessmentppt3222016.pdf

Biographical Sketch

Andrew Pass is the founder and CEO of A Pass Educational Group, LLC an educational content development company that partners with organizations to develop customized content. A Pass’ more than 1,500 associates include writers, editors, subject matter experts, instructional designers, artists, and production team members. A Pass has deep expertise developing performance based tasks and assessment items, amongst many other things. When A Pass partners with an organization, they take their client’s specifications and objectives as their own and become their back office content development center.