June 11, 2019

Line Dempsey: Welcome to our podcast, Regulation Matters: a CLEAR conversation. I'm your host, Line Dempsey. For those that do not know me, I am currently the chief compliance officer with Riccobene and Associates here in North Carolina. I'm on the CLEAR board of directors as well as the current chair of the National Certified Investigator Training committee and vice-chair of the Annual Conference Program Committee with CLEAR, which our Annual Conference happens in September of each year. As many of you may know, the Council on Licensure, Enforcement and Regulation, or CLEAR, is an association of individuals, agencies and organizations that comprise the international community of professional and occupational regulation. Our podcast is a chance for you to hear about the latest and greatest in our community. Today, I'm joined by Sarah Wennik. She is manager of content development with Pearson VUE and chair of CLEAR's Examination Resources and Advisory Committee. We're glad to have you with us today. Welcome.

Sarah Wennik: Thank you so much. I'm grateful to be here and very excited to be here and speak with you today.

Line: So that's great. We really do appreciate you joining us. Today, the topic that we'd like to talk about is the role of a content developer in the context of professional and occupational license examinations. So I guess, let me start by asking, for those that may not know, what does the content developer do? Are they actually different than psychometricians? How does that work?

Sarah: Sure, that's a great question. And honestly, it was a little over a decade ago, I found myself asking that exact same question. It wasn't necessarily a job path I was familiar with, although it's turned out to be one that I enjoy tremendously, and I want to share that enthusiasm and excitement with everyone. But again, it's not necessarily something when someone asks you 'what do you wanna be when you grow up?' -- most people probably aren't gonna answer content developer or even necessarily psychometrician for that matter. I think, simply put, if you want a great way to remember it: psychometricians = numbers; content developers = words.

So we work together; we're really part of the same family. Psychometricians and content developers are gonna work in partnership with each other. The content developers focus really on the linguistic content of examinations themselves. So down to the question level, the questions themselves, but also how the questions work together within an exam. And we really work closely with test sponsors and subject matter experts on those test questions from really the genesis of the exam at the content outline or content blueprint to developing items, writing them from scratch, or revising and improving items, re-validating items. And then after the item or the question has been finalized, we work closely throughout the remainder of the exam development process with psychometricians to assemble all of those test questions into the exam that your candidates are ultimately going to take. So, that's the nutshell, right? And then that's a little broader context of what content developers do.

Line: Gotcha, it seems very similar in the world of website design. You've got your people that design the website and then you've got the usability people that look and make sure that people can actually use it. So they work hand-in-hand to come up with a final product that's valid and reliable, right?

Sarah: Absolutely, and so we're really a partner. I think I consider content developers a partner not only of the psychometricians (we work closely with them), but we're also partners with the test sponsors. And the content developer is making that nexus, much as you've described with usability, to make sure that the questions are certainly performing from a statistical perspective the way we'd expect, but how do you operationalize that, right? How do you take numerical data and actually make changes to test questions or develop additional test questions to see those numbers in practice to make sure you have a test that hits all the numbers? Well, again, we have to do that with words and with content.

Line: So that kind of leads me into, I guess, my next question. What kind of skills maybe would make someone a good content developer?

Sarah: Absolutely, I think being a content developer is an unusual set of skills because you almost need facility in completely divergent areas to be a really successful content developer. And what I mean by that is, you really have to be comfortable with both kind of stereotypically introverted tasks as well as pretty stereotypically extroverted tasks. And if that wasn't enough, you also need to be pretty good with some technical aspects in terms of software and using technology and also -maybe these two are a little more related- comfortable with using data to get things done and to work with others.

So I can break those down a little bit to give some examples. When I say introverted work, I think these are the folks who like to work individually, maybe reading test questions. They're gonna be editing grammar, spelling, using terminology consistently, maybe creating or applying a style guide to a set of questions so that, again, questions are formatted similarly when they're asking about similar topics, that you're referring to your licensee in the same manner or to other aspects of your testing program, right? We don't want noise in the way that the question is designed. You don't want candidates thinking, 'Well, they refer to the licensee this way in one question here, but now they're referring to it differently here. Does that mean something? Am I supposed to be seeing something different in that question?' I mean, obviously, if there was an intention behind that, sure, but if it was just simply a lack of consistency, that noise is taking away from the candidate's concentration and ability to demonstrate their competence, right?

So we wanna avoid extraneous inconsistencies or confusion within the items. And so when I talk about the "introverted" part of the job, I'm talking about that kind of work - really being able to spend solid amounts of time looking at test questions and doing that kind of reading, editing, grammar. And that seems like you can kind of conceptualize it--maybe a person, maybe someone you know, maybe yourself--that's really your wheelhouse, right? You like to sit there and edit text, like that copy editing piece.

Well, that person is not sufficient to be a content developer, and here's why - because that work happens, but the next piece (or, I don't know that it's entirely sequential; kind of happens simultaneously sometimes) is you need to be able to then take that work and present that in front of your test sponsor and your subject matter expert group. So often--and again, I spent many years as a content developer; my role now is really a manager of content developers--but my folks are out there working directly with the client, recruiting SMEs in many cases, setting up meetings, organizing who's gonna be there, facilitating those meetings, presenting test questions for hours and hours in front of exam committees, right?

And so, you need to be comfortable reviewing those items on your own, both pre- and post-meeting for those things that we talked about: consistency, grammar, spelling, etc. But you need to be able to hold that room and have the ability to facilitate amongst different groups. Some of my folks work entirely within one market segment, as we call it, or one group of exams. We have folks who focus entirely on insurance examinations, for example. But many, many content developers have the opportunity to work with a variety of different professions. So that extroverted person not only has to be out there in front of the group, they actually have to be comfortable leading from a place of their own expertise.

I'm not the expert because I'm the best real estate agent; I'm not a real estate agent. I'm the content developer and I'm gonna sit with a group of people who are really esteemed in their profession who have done this job for, often in many cases, decades. They really know all there is to know about real estate, and that's a powerful group of people to be with. The content developer is going to use their leadership, their ability to facilitate, to get that group to work together. Even though the content developer themselves is not necessarily a subject matter expert in real estate or whatever profession, they are going to be the ones who can lead that group and really design the work that they need to do to understand what the goals of the meeting are and to, again, lead a group of diverse professionals who have strong opinions in many cases, a lot of knowledge, to get those tasks done.

So we've got that introvert who's gotta be doing that work in the background. We've got that extrovert out there in front of a group of very powerful and knowledgeable people. And then you need to be someone who's really comfortable using technology. Again, oftentimes these are proprietary systems, so you may have worked for one organization, but you go to a different organization as a content developer, they'll have a different item banking system or testing system. Typically you're also using all of the technology that comes with a normal office environment, your word processing, your Excel or spreadsheet programs. And you need to be able to do that in front of people. I don't know if you ever had anyone watch you type? I don't know, maybe this only happens to me, but somebody watches me type and suddenly I can't type anymore. Well, again, I had to get over that in my job as a content developer because you're literally editing questions in real time in front of people who are absolutely watching you type and watching you spell. So again, there's a certain comfort that you need to have; there's a vulnerability there, right? You need to have comfort and be willing to put yourself out there.

And then the final piece is really being able to understand the data that the psychometricians provide and work with the subject matter experts to take action on that data. So again, you need to understand it as a content developer, but you also need to be able to make that understanding available in a way that the subject matter experts can use, can utilize. It's not just a data dump.

Line: That sounds somewhat challenging. You really have to be a Jack or Jane of all trades to kinda cover all four of those bases. That's pretty amazing. I guess, and you kind of touched on a little bit with some examples, but maybe go into a little bit more detail of what tasks the content developer actually performs.

Sarah: Sure, and again a little caveat here, this isn't to say that only content developers perform these tasks or only psychometricians perform certain tasks. I would say there's variability based on the organization, the structure of the organization you're working with, whether you are a larger testing company or a smaller boutique testing firm. So there may be different degrees of overlap.

What I can do is describe some of the things that we do here as a team of content developers with the caveat that other organizations may structure their roles a little bit differently and again there is often overlap between, in some cases even here, what our psychometricians do and what our content developers do. But really your content developer is going to be shepherding your exam from its birth really to its deployment out into the field to your candidates. So typically, both psychometricians and content developers will work on developing your -we call it either- content outline or content blueprint, working on the structure of that exam that might be as the result of a job analysis (typically it is, or there may be other ways that content outlines are developed). And really, keeping an eye on how your item banks match up against that content outline is a major responsibility of the content developer.

Great, so your blueprint will say you need a certain number of items in each area, and one of your content developer's major responsibilities is keeping up with that and ensuring your item pools have enough items to be able to build as many versions of a test that you need, right. So you may need five items in a particular content area, but if you wanna build multiple forms of that exam or go to a type of pool testing, you're gonna need more than just five, obviously, for that. And so your content developer will perform what we often call a gap analysis, and that really is the analysis of what items do you have in the bank, how many questions in each area, how many of them are truly different.

So one of the things the content developer brings to the table is really that, again, with the words; it's not just numbers, it's not just 'Oh, I have five items.' You could have a situation where really three of them, they're really asking the same thing. So ideally you don't want all three of those on your same exam form, so you need even more items. The numbers made it look like you were fine, but actually when you look at the content of those questions, it's not enough 'cause it's three of the same, three of the same item. Or, again, a test question might give away the answer to another test question. We call those enemies in my world here. You don't want enemies on the same test.

Line: [Laughter] I call those "freebies!"

Sarah: [Laughter] That's a freebie, yeah! When you're a good test-taker, you're gonna pick up on those things and again, it makes you feel really good as a test-taker to see that. But certainly from a test sponsor's perspective, you want someone to have demonstrated their competence because they actually know the answer, not because they're super-savvy test-takers and saw that the answer was three questions ago in the other question.

So again, your content developer will either create that gap analysis; it may also be created from a psychometrician’s work. But they're going to operationalize that for your test development meeting, so they typically provide item writing assignments based on a combination of that gap analysis and that in-depth reading and knowledge of your item pool, right. Again, we don't wanna just assume, 'Oh, I have five questions; I'm fine for this section,' because some of them might be cueing or some of them might be the same question.

And then being able to provide that facilitation and coaching during the meeting, right, so giving out really precise and useful item writing assignments so that the SMEs... The SMEs we work with are so fantastic. They want to do the job right. It's really our responsibility to make sure we're setting them up for success so that we can achieve those goals that we've set for that meeting, whether it's a certain number of items (it's often a certain number of items), but what type of item. There may be additional considerations. Some programs really look at the cognitive level of the item, right. Is that a recall question, is that an application question? If everything's just recall --how many of these do you need in your hair salon, how many towels do you need?-- that's a recall question. I need to know how many towels or I need to know this, but often that's not how candidates or licensees approach their job. It's not through just a memorization process. So in a realistic situation, can you remember the information that you need to recall when you're in that situation? A good content developer is gonna help your subject matter experts understand the difference between those kinds of questions, as an example, and be able to produce those kinds of questions when necessary in your item pool.

So there's coaching, there's some design of assignment there. And then once all the items are reviewed and re-validated, there's a whole form assembly process that content developers typically participate in. So that's the process of selecting which of those questions are the lucky ones that get to go on your exam form for this administration and for this testing window. And again, they're gonna be reading those test forms making sure they're meeting not only the numerical specs but also the breadth of the content. So you don't want to meet your numerical specs but really have all five of those questions essentially asking about the same topic area.

So, content developers are gonna have an intimate knowledge of the pool to ensure that, again, not only are you meeting the content specs and what the psychometric specs are (the cut of the exam is one example), but also are you covering broadly the topics that you need to cover.

I'm gonna plug -I hope this is Okay- I'm gonna plug the annual conference in September, because one of the things content developers do, really where they shine in terms of this nexus between psychometrics and clients and SMEs, is the ability to coach groups of SMEs on revising items with poor statistical performance.

Right, so you get these numbers from your psychometricians and they say, 'Okay this item has a poor point measure or a point by a poor point bi-serial, or the P-value on this item doesn't really work,' or there's other psychometric terms they're gonna say. Well, as a test sponsor or even a subject matter expert, what do you do with that information? I can tell you, 'Okay, the point measure's just not gonna cut it; it doesn't meet our standards,' but how do you actually address that item to make it better? How do we make a change? We have some information about how this item is doing with candidates; now what do we do? I'm not going to tell you now. You have to come to our session on Thursday, September 19th at 10:30 am in Minneapolis. It's called "What Do These Numbers Mean, and What Can I Do About It?" And it's gonna be great. There'll be a presentation, but we'll also have small groups with a number of amazing table coaches, some of whom are psychometricians and some of whom are content developers. And you'll actually get to work on example items with some of these statistical problems and practice re-mediating them. So again, that's very much in the content developer wheelhouse. And so if you really wanna experience that, you can come to our session, and it'll be a great example of the work content developers do.

Line: Well, that sounds exciting. Obviously, we've talked a lot about content development, but now you just mentioned too at the presentation, you're gonna have psychometricians there as well. If you are sponsoring a test or you had a test sponsor that was looking at either a content developer or a psychometrician to work on their exam, should they choose one over the other? Is one a better choice?

Sarah: Well, I truly believe that you need both to really have a successful exam. You absolutely need the expertise, the knowledge, the analysis that a psychometrician is gonna provide on those exams. That really provides a lot of the fundamental defensibility for your examination program, and they provide again a great deal of data that your content developers and then your test sponsors and your SMEs are gonna work with to continually improve that exam again, to keep the defensibility, the fairness, keeping those forms balanced, keeping the reliability of the forms in place. So again, I'm gonna have to say you really can't choose one or the other. You really want both of those excellent professionals working on your exam program.

Line: That's great - good answer. [laughter] I appreciate you being a part of our podcast today. I think this has been a great conversation, and one of those things that I think will spark some interest towards your presentation that you guys have in September. But it's always a wonderful opportunity to kinda get together and share and learn from each other. So again, thank you for speaking with me today.

Sarah: Absolutely, it's been a pleasure.

Line: And I also want to say a special thank you to our listeners. We'll be back with another episode of Regulation Matters: a CLEAR conversation very soon. You can subscribe to our podcast on multiple different avenues: Podbean, iTunes, Apple Podcasts, Google Podcasts, Google Play, Stitcher, Spotify or TuneIn. If you've enjoyed this podcast episode, please leave a rating or comments in the app. Those reviews help us improve our ranking and make it easier for new listeners to find us. Tell your friends about it and also feel free to visit our website at www.clearhq.org for additional resources and a calendar of upcoming training programs and events. An example of that is our Annual Educational Conference in September. Again, finally, thanks to our CLEAR staff, specifically Stephanie Thompson, our content coordinator and editor for our program. I'm Line Dempsey, and I hope to be speaking to you again very soon.

The audio version of this podcast episode is available at https://podcast.clearhq.org/e/content_developer/

Home  |  About  |  Membership  |  Resources  |  CLEAR Learning  |  Awards  | Events
Join  |  Contact  |  Login  |  eNews Signup  |  Communities by CLEAR  |  Privacy Policy

Copyright © 2020 CLEAR
Please read the following disclaimer.  |  
Website by Nicasio LLC and CLEAR
108 Wind Haven Dr., Ste. A
Nicholasville, KY 40356
(859) 269-1289
Fax: (859) 231-1943
Powered by Wild Apricot Membership Software