NBPTS v3.0 Selected Response Item Review: A Trip to The Dark Side

I field-tested the National Board v3.0  AYA Science Constructed Response and  Selected Response Assessment Center components. I was then invited to the Item Review, to happen here:

Wow.

Know thine enemy, I’d been told. Besides, if I didn’t go check this out, how would I ever be able to comment with credibility on the process?

I went.

After arriving in Austin after midnight, I slept really fast, had a nice hot hotel breakfast, and hopped a shuttle for the meeting center.

The center was clean, neat, office-ish, and not busy aside from 2 other groups of NBCTs doing reviews in their subject areas. I’d have more photos, but the nice man at the desk stopped me as I was about to take a photo of the sign saying, “No unauthorized photos inside building.”  Disclaimer: I was given permission to describe the process, but not content or the test.

Once inside, we were fed a snack and then had about an hour of orientation to the process and the test itself. We got a briefing on equity, including the non-use of gender-specific pronouns. The test had already been vetted by an equity committee, but we were asked to monitor equity as well. We were given another snack and sent to a conference room with two nice Pearson employees, both former science teachers. Both had worked for Pearson for a few years and clearly had learned much about testing validity and reliability. They had all the field test results, complete with p-values and other reliability measures I can’t discuss. I can say that I was comfortable with these data.

Six NBCTs and I, all AYA/Science  – Chemistry, went through the test questions, on at a time. There were a lot of questions. Some were familiar, from my field test. We scrutinized wording, including terms that were either confusing, misleading, and in a few cases, outright incorrect. We had a nice lunch, then worked right up until 5:00PM (I had a plane to catch to be back in school the next day – whew!) I was, as usual, totally impressed with my NBCT colleagues. They were knowledgeable, professional, and a delight to work with, because, well, NBCTs.

There could be no mistakes, bias, misinterpretations, arbitrary answers when we finished. We worked every problem, including all the distractor answers, re-phrased, edited, and vetoed. We considered the field test results. We were heard. I left the process at the end of the (long) day with confidence that this test measures what a high school chemistry teacher should know.  Because I’ve scored, I know that the scoring process is also run professionally, with the utmost validity and reliability.

Am I happy that Pearson won the bid for scoring National Board assessments and the portfolio? No. No, I am not. As a scorer, and now as an item reviewer, I do have respect for the process and do not question any part.

As a tease, the trainer suggested that Component 3, Teaching Practice and the Learning Environment, will have 2 videos. I’ll find out soon, as I’ve been chosen to field test that component, too.

%d bloggers like this: