April 13, 2021


Line Dempsey: Welcome back to our podcast, Regulation Matters: a CLEAR conversation. I’m your host, Line Dempsey. I am currently the chief compliance officer with Riccobene Associates Family Dentistry here in North Carolina. And I’m also the chair of CLEAR’s National Certified Investigator Training Committee.

As many of you are aware, the Council on Licensure, Enforcement and Regulation, or CLEAR, is an association of individuals, agencies and organizations that comprise the international community of professional and occupational regulation. This podcast is an opportunity for you to hear about the important topics in our regulatory community.

With today’s episode, we want to acknowledge the important contribution of two CLEAR members. Joining us today, we have Steve Nettles and Jim Zukowski, who have recently stepped down from a long tenure on the editorial board for the CLEAR Exam Review journal. We thank you for your service and dedication over the years, and we’re super glad to have you with us today. So, welcome.

Jim Zukowski: Thanks, I'm glad to be here.

Steve Nettles: Yes, likewise. This is quite an opportunity to review 30 years.

Line: Absolutely, well, we're super glad to speak with you, and also let me thank our listeners for joining us today. You know, I hope our listeners have had a chance to benefit from CLEAR Exam Review; I know I have over the years. But let me set the stage for our discussion today. CLEAR Exam Review is a bi-annual journal that has useful discussions on current licensing examination issues, geared toward our broad audience that we have at CLEAR.

Going back to CLEAR’s archives, the journal began in the 1980s with its pre-cursor, the National Clearinghouse on Examination Information. Then, 1990 was the first issue of our CLEAR Exam Review. Steve, you joined the editorial board with Volume 3, No. 1 in Winter 1992, and Jim, I understand you joined in Winter 1994. Take us back, if you will, to those beginnings and how you first got involved. And let me start with Steve first and then we'll go to Jim. So, bring us up to speed.

Steve: OK. Well, thank you, Line. I remember back quite a few years, but at that time, when I first heard about CLEAR, I was working at Educational Testing Service. Real close to my office was Ben Shimberg. And Ben, as you know, was a kingpen in the formation of CLEAR to begin with. So, between Ben and my boss at that time, Mike Rosenfeld, they encouraged participation in CLEAR. And when I left ETS and started working with what was then Applied Measurement Professionals, which is now PSI, in the late eighties, I started going to CLEAR meetings. I enjoyed the opportunity that was there to work with colleagues in the credentialing testing industry and get a different slant from the regulators within the CLEAR organization.

Jim: Well, my first initial outing with CLEAR, it began back in the eighties. I had just moved in from being at the University of Texas, and I started with the Texas Department of State Health Services. And I started initially dealing with the emergency medical services licensure and testing program, and shortly I became involved with the Professional and Licensing Division and Certification Division for the Texas Department of Health, and it included a wide array of licensing programs.

And my boss had attended one of the first CLEAR conferences that was organized. And when he came back, and he said, ‘Jim, you need to start getting involved in this organization. They do have a track that involves examination section.’ So, I first started attending the CLEAR conferences when they were in Orlando, I think, back in the late eighties. And I have tried to go to CLEAR ever since then.

And the Examination Resources and Advisory Committee has been a real blessing for me, because I had a chance to experience dialog with a variety of people from a variety of services in a variety of states. And I met Steve through CLEAR and people like Kara Schmitt and Barb Showers, our friends in the state of Colorado, and the national testing services, such as ETS, with Mike Rosenfeld, and some of the other people that just got me actively involved in the organization. I learned so much in terms of how did you deal with the examination components of the regulatory program that I was in charge of.

Line: Well, it's quite a lot of history between the two of you guys, and I know you've seen a lot of changes over the years. What would you say some of the biggest changes, and maybe innovations, that you've seen in the testing industry since then? And I know it's probably fairly big. Jim, let's start with you on that.

Jim: I think the biggest change has been going from a strictly pencil-and-paper examination program to computer-based testing. And I first remember when I was with the Emergency Medical Services program, we did Scantron sheets, and we didn't even have the ability to grade the written examination. So, we had to contract with the University of Texas and take all of our examination sheets over there and get them graded through their main line computer. And, that's how it started, and over the years, we've just gone through computer based testing, and it's been a godsend. There are some additional costs, but it's helped us examine more people through a wide variety of venues and it's made the whole examination process a lot easier in terms of manpower.

Line: Steve?

Steve: Well, yeah, I'll start with the changes, and the changes and innovations are some overlap. But for me, one of my jobs before I came to ETS was working for the federal government. I was in charge of grants for a five-state area in the mid-atlantic region. One of my jobs was, when the Uniform Guidelines came out on employment testing from the Department of Labor, Department of Justice and the Civil Service Commission; it was 1978. I had the wonderful job of training personnel departments in those states on the Uniform Guidelines. And when they first came out, there was controversy whether or not they applied to credentialing exams. And finally, I think everybody agreed it did, so to me, the big change was now credentialing exams were held to the same standards and requirements that employment tests were held to.

The big change for me was when I first started working with clients and a lot of the clients that we're developing exams for had never done a job analysis, and that was one of the basic requirements. It also targeted things besides job analysis or practice analysis, but you couldn't just come up with a passing score. You had to do passing score studies. There was a lot of the discussion about content validity and other types of validity, and whether empirical validity studies applied or whether content validity were sufficient. For the most part, content validity was used to support the validity of exams. And because of that, there was a lot of documentation requirements that gradually evolved over the years, especially in the eighties and as it got improved in the nineties. So, to me, that was a big change. So now, we had requirements we have to meet, and you could be legally responsible for that. Hopefully you never did - that was one of the most unenjoyable activities, being an expert witness in support of an examination.

The innovations - I agree with Jim. The big one with CBT. But I would take it back a step earlier to the computers. When I first switched over in the eighties and changed jobs to work with AMP, we had one that we called the bed frame. Personal computers were just breaking news then. As they got more and more power powerful, we got to apply some of the more complicated testing models to go along with this. The first one I remember was item banking. In my very first career - that was back in the state of Illinois, back in the seventies when we were doing civil service testing. We had card files, and each card had an item on it. When you wanted to develop a test, you would assemble a page of the test with the cards and take a photocopy of it.

Line: Wow!

Steve: Yeah, it was horrible. But that was nice when the first application we did was item-banking on the computer. So, it was nice to have the items there, especially made the tests look real nice when you could print them out and make copies that way.

The other thing I noticed as the innovations were passing score studies were pretty much non-existent, or they were done on the old method of, was a relative standard a statistical standard, where they used the general rules like one standard error or half a standard error below the mean. It took a while to convince many of my clients that this was not a good way to do it, and we gradually got them into more of the newer ones: Angoff, and later on, Angoff continued to be very popular, but there are some new methods introduced.

The innovations over the years, as I looked at 20 to 30 years of abstracts in the CER, I noticed there was a lot of talk about item innovations. As computers were allowing us to do stuff, we did videos, automated item generation. I never did the automated item generation - I thought that was a little difficult to do for the type of clients I had, but it evolved.

And then of course, candidates will be candidates. As things happened and the CBT came out, the cheating got more and more popular so there was a constant battle between the cheaters and the test providers.

And then the last thing, I think the innovating thing that I noticed was there's more and more talk about continuing competence assessment, in the later years. You were licensed once, and, you know, 20 or 30 years later, what do you remember? I probably could have used some continuing competence as I got later on in my career. But those are some of the things that I brought up that I think were the changes and the innovations.

Line: Right, that makes good sense. It kind of leads to, obviously, some challenges that were overcome thanks to technology as things improved along the way. I guess, looking back, and, Steve, if you'll continue, kinda what were some of the biggest challenges the testing industry has actually faced, in your opinion?

Steve: Yeah, as you said, they sort of overlap with the innovations. As we innovated more and started implementing the CBT exams, of course, the candidates evolved. If you can call it test security, or I like to call it candidate misbehavior, improved. So, it was a constant battle over who was going to maintain the validity of the exams. Surrogates was a big issue in both paper and pencil and CBT, and some of the biometric standards helped us work with the surrogates.

Item harvesting was always a problem with a group of candidates memorizing items that they could do it on paper and pencil before it was CBT, and then, there was a push to do it on CBT. But they got more clever and started using more sophisticated technologies like little tiny cameras. It was a constant battle trying to figure out what's the next thing we're gonna have to fight against, and especially as “spy technology” became more and more available on the Internet.

Then as more and more state organizations started using national or multi-state exams, that created an issue where we would have to include a lot bigger sample in our job analysis or practice analysis and try to find commonalities among the states that everybody could agree on that should be on the test. We had a lot of interesting discussions about that.

A little bit of the international exam development- that was popular for a while. I guess it still is where it goes up a little bit with the Canadian companies. The French Canadians and the translations - that became an issue. You couldn't just do a straight translation. Then there was a whole bunch of research and technology improvements to get higher quality translations.

Then, I think it was maybe in the nineties, early two thousands, there was a big push for innovative item types - how to get those into your exams and how did they mix if you were still using multiple choice questions. How did that all fit together? The OSCEs, the practical exams. Yeah, Jim, you’re familiar with those.

Jim: Yeah, can I elaborate on those practice exams? One of the challenges that we had over in Texas was the practical exam for a massage therapist. And the laws, as they existed in the state at that time, required us to do a written as well as a practical examination to license massage therapist in the state of Texas.

And first of all, we had to overcome the stigma of what actually was a massage therapist. And through a lot of education, we finally convinced the community that this was a legitimate occupation. And our committees stressed the fact that we needed to do a practical examination to test whether or not these people can actually do the different kinds of massage techniques, as well as administer a written examination. And we worked in partnership with Steve’s organization, Applied Measurement Professionals. And we did actually administer a practical examination whereby a candidate would go into the room, they would have a massage table and a client, and the session, which was very short, was videotaped. And once the sessions were videotaped, it would go back to a separate room that had a bunch of skill examiners in it, that would review the tapes. And we would get at least three reviews of each videotape before it was decided whether or not that person actually practiced examination according to some criteria that were pre-established about what constituted acceptable performance of a particular massage therapy technique.

And this was extremely labor intensive. You know, we had to contract with various hotels to get the hotel space so that we could strip down hotel rooms and put in massage therapy tables and do the videotaping. And we used to do that three times a year. And it just wore us out. We had to do it on the weekends, and we would test hundreds in each session. And thank God, they finally abolished the practical examination and went strictly with the national examination, which certainly cleared a lot of the anxiety and a lot of the work involved in administering the examination component for that particular program. So, I can only remember some of the horrors that happened during some of the sessions that went on. But that was part of the challenges we faced in the testing industry at that particular point in time.

Line: [laughing] I thought for sure you're going to say that you volunteered to have the massage done.

Jim: Well, in some cases, we had to because either they couldn't find a model or something happened and the particular model that we were going to use sometimes wasn't available. Or, you know, a lot of times we would have two massage therapy candidates just rotate back and forth thing being actually a client as well as the practitioners. But, it was a challenge; that's for sure. But it was successful.

Line: Alright, well, I guess, Jim, if you had a crystal ball in front of you, what do you see for the future for the testing industry itself?

Jim: Well, I think the direction is going more towards internet-based testing, and I have some reservations about the security involved in all of that. But I think in terms of conducting the examination component quickly, affordable, and by the means to assess the competence and achieve the results within a very short period of time, I think that's the way it goes.

For the future, I can't think of anything else that would be technology advances for us to go into another direction to do. Otherwise, I can't think of anything else except go back to some traditional approaches to testing that I don't think the testing industry would support that. That's the way I think it's going.

Steve: And I'd like to elaborate on Jim's topic about the online testing. To me, I'm not a big fan of it either. The proctoring issue is always something that I could never feel real comfortable with. But one of the more practical topics, I think, was brought up at a couple of CLEAR meetings was the interfaces for online test administrations. And everybody is at home and taking the test on their own equipment - they all have different equipment. Does the interface work for, say, one of the basic tenants of valid test scores is that there's commonality, that everybody has the same experience. And I wasn't convinced that you will, but you know, I could be showing my age there.

But there's one other thing I wanted to mention about challenges was when ADA jumped in, and it made the whole accommodations issue. And that again, gets back to the commonality of testing. Because if you have a candidate with disabilities, you needed to provide possibly extra time or a different format for people with visual disabilities. So again, that's going to be a continuing problem, I think.

I'm gonna go way out on the future of the testing industry, I think as artificial intelligence adapts and comes in - this, you know, may be 20 or 30 years out, or it may never happen. I won't be here, so it won’t matter. I was thinking about this, what would be really neat is virtual reality. A long time ago, when I was working, I can't remember where it was, we were developing an examination for people that take emergency calls from the police department and fire departments. And we set up a little thing that in one channel - they had to listen to headphones - so the one ear would be the people calling in; the other ear would be other complications from other apps. You know, so the controller (I can't remember the name of the position), but the people who handled those calls had to be, you know, listening with both ears to two different things. Virtual reality would make that a lot easier – you could put the person in the vignette, sort of like some of these police simulations or the military going into a burnt out or a bombed building in some other country and trying to find terrorists. And what's your first reaction - do you shoot or is this a civilian or is it a terrorist? So, I think virtual reality may come into that in the future.

Jim: I think the pandemic has also brought to light in the industry and certainly some of the testing programs I'm dealing with right now is the fact that, because so many businesses and industries have been affected by the pandemic, it’s going to force us to go back and take a look at some of the job descriptions that have been traditional for many of the occupational programs. I'm dealing with the people who are in the food service industry, and in Texas, thousands of establishments have just closed because of the pandemic, and in those that have remained open with limited seating capacity, employees have just learned to take on so many additional skills and responsibilities. Really, when you actually talk to them, in terms of what they thought they were going into versus what they're actually doing are two separate things. So, I think a lot of the testing groups are going to have to go back and get consensus: what actually are some of the new job responsibilities for certain classes of occupations?

Steve: Yeah, it'll be interesting. COVID sorta threw a monkey wrench in everything.

Line: Absolutely. Well, it's been a great conversation. You know, we want to thank you, Steve and Jim, for volunteering your time and expertise over the many years to bring this relevant and timely information to our readership. So, thank you for speaking with us today.

Jim: You’re more than welcome.

Steve: Yeah, you’re very welcome. It was a great opportunity to, number one, connect with Jim again - we used to see each other a couple of times a year - but also just sort of reminisce about one's career and how big a part CLEAR was to creating a network of other like-minded psychometricians.

Jim: I agree with Steve wholeheartedly. One of the things that CLEAR used to have that I wish they would think about starting again is they used to have a column entitled “Testing Across the Nation.” And I took over that responsibility many years ago, but people such as Kara Schmitt who was in the state of Michigan, Barb Showers in the state of Wisconsin, and, of course, I was here in Texas. We would actually poll testing experts around the country who were members of CLEAR about what was going on in their state or in their particular testing program. And we would assemble that information and write a column for each issue of CLEAR Exam Review about what was going on across the country. And it was a very valuable tool in terms of finding out exactly what happening, and you had, at least, a contact person to go to if you had further questions about that particular program. And it was a lot of fun. That particular column was extinguished a few years ago, but maybe the active CLEAR committee members could think about starting a column like that again.

Steve: Yeah. I agree. That was one of the benefits of all the different columns in the CER - there's all sorts of topics brought up, and people presenting papers that Jim and I and the other folks would review. And it was a great educational opportunity to see how things were done by somebody else. I think it was a great program when CLEAR decided to start sharing this information way back, I guess as you said, in the late eighties.

Line: That's a great idea. You know, in the interim, we might be able to utilize our CLEAR Communities, which I'll talk about that in just a moment. But I just wanted to thank you guys for being a part of CLEAR for so many years. And it certainly has been a pleasure to talk with you all today.

You know, I also want to acknowledge Adrienne Cadle and Sarah Wennick, who are joining the editorial board for CER. Work is currently underway for the Summer 2021 issue. We wanna thank Adrienne and Sarah for the new perspectives that they’ll bring and we look forward to the continued success of CER. A reminder to our listeners that CLEAR members can access issues online, and subscriptions can be purchased through CLEAR’s publications page on the website.

Also, I want to take a moment to thank our listeners for tuning in once again for this episode. We invite you to continue the conversation through our CLEAR Discussion Forum, which is what I was mentioning just a moment ago. This podcast will be posted in the CLEAR Communities. Members can reply to the post with their comments. And, you know, I'll pose a couple of ideas, kind of seeds to plant. What challenges and innovations do you see in the future for licensing examinations? Maybe, you know, what testing and examination topics would you like to hear discussed at CLEAR events? We'd really love to continue this conversation on CLEAR Communities.

And we'll be back with another episode of Regulation Matters: a CLEAR conversation very soon. If you're new to the CLEAR podcast, please subscribe to us. We're available on a lot of places: on Podbean, Apple podcasts, Google podcasts, Stitcher, Spotify, TuneIn, Alexa for Amazon Echo devices, iHeartRadio, Player FM, ListenNotes, and Pandora. And I think Stephanie keeps adding more every time we do this. So, if you've enjoyed this podcast, please leave a rating or comment in the app. Those reviews help us to improve our ranking and make it easier for new listeners to find us.

And also feel free to visit our website at www.clearhq.org for additional resources as well as a calendar of upcoming online programs and events. Finally, I'd like to thank CLEAR staff, specifically Stephanie Thompson. She is our content coordinator and editor for our program. And once again, I’m Line Dempsey, and I hope to be speaking to you again very soon.


The audio version of this podcast episode is available at https://podcast.clearhq.org/e/CER_retrospective



Home  |  About  |  Membership  |  Resources  |  CLEAR Learning  |  Awards  | Events
Join  |  Contact  |  Login  |  eNews Signup  |  Communities by CLEAR  |  Privacy Policy

Copyright © 2020 CLEAR
Please read the following disclaimer.  |  
Website by Nicasio LLC and CLEAR
108 Wind Haven Dr., Ste. A
Nicholasville, KY 40356
(859) 269-1289
Fax: (859) 231-1943
Powered by Wild Apricot Membership Software