According to NCEES, "After each pencil-and-paper exam administration, every exam undergoes extensive statistical analysis to determine the minimum level of performance required for entry into the profession. This rigorous process helps ensure an equitable licensure system for examinees while upholding the standards of the profession."The Control Systems cut score meeting.
The PE control systems exam has a new specification for this exam administration. When that happens, the subject matter experts meet in person to determine the cut score and set standards for future exams. This is usually one of the last things that takes place in the exam scoring process. Results typically follow within a week after that. This year, the Thanksgiving holiday is the last week in November. This will, very likely, push the control systems cut score meeting to the first week (or so) in December, which will, in turn, probably push the results to the second week in December.
You are correct. However, the various committees (or whatever they are) don't set standards for future exams after each administration and, more importantly, they don't necessarily meet in person. It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results. Correct me if I'm wrong, @RBHeadge PE.To me this reads that a cut-score is determined for every administration, not just exams with new specifications.
Yeah send it to me. I'm bored and just interested in their process. I updated my post, btw, with some more information/questions.You are correct. However, the various committees don't set standards for future exams after each administration and, more importantly, they don't necessarily meet in person. It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results. Correct me if I'm wrong, @RBHeadge PE.
Either way, if you'd like something more definitive than just my (and RB's) word, I can send it to you. I'm new at reading the tea leaves, but it's been very consistent over the last 10, or so, years. Just let me know.
Typo on my part. Edited.However, the various committees don't set standards for future exams after each administration
My employer blocks EB on my work computer, so I may or may not end up sending it this afternoon on my phone. If all else fails, you'll get it this evening, after I get home.Yeah send it to me. I'm bored and just interested in their process. I updated my post, btw, with some more information/questions.
So:It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results.
These guys do meet in person?The PE control systems exam has a new specification for this exam administration. When that happens, the subject matter experts meet in person to determine the cut score and set standards for future exams.
I agree. I think they're the sameĀ atĀ each administration. I could say I know that for a fact, but I won't say it.Now I do think the exam are the same within each administration. Like all Oct '19 Power exams are exactly the same. Something you said made that seem unclear to me. I'm also wondering how that will change with the transition to CBT.
Yes @Chattaneer , each administration of every exam has a different cut score. But the cut score is determined based on the component questions. Generally speaking for P&P exams,Ā the weighting for each question is determined following the first administration an exam after is adopts new specification, at a "standard setting" or "cut score" meeting. That meeting is held in person. A new "cut score" meeting isn't needed for subsequent administrations that use the same specification.According to NCEES, "After each pencil-and-paper exam administration, every exam undergoes extensive statistical analysis to determine the minimum level of performance required for entry into the profession. This rigorous process helps ensure an equitable licensure system for examinees while upholding the standards of the profession."
To me this reads that a cut-score is determined for every administration, not just exams with new specifications, but I could be wrong.
Step 4 of their process: "When the analyses and reviews are completed, NCEES changes the answer keys as necessary. The passing score and final correct answers for each exam are then used to score all answer sheets. A percentage of the answer sheets are manually verified, and the results are compared to the machine score to ensure accuracy."
Again, this leads me to believe that the cut-score has to be determined after each administration.
The questions come from a question bank. I wouldn't ever expect to see the same exam (questions and order) offered twice.Maybe someone could answer me this, I know that there are multiple exam variationsĀ for different exam sessions. For example, there are multiple morning session exams (like morning exam 1, 2, 3, etc.)Ā and multiple afternoon session exams. Does anyone know if there are a finite number of exams, or is there a question bank that questions are pulled from to generate an exam session?
Maybe someone who has participated in PAKS could enlighten me.
Oh that brings up another question. On the front of my exam it said "all questions are weighted the same" which goes against everything I've been told.Generally speaking for P&P exams,Ā the weighting for each question is determined following the first administration
You'll have to speak slowly and use small words with me, I'm not the sharpest crayon in the shed. lolYes @Chattaneer , each administration of every exam has a different cut score. But the cut score is determined based on the component questions. Generally speaking for P&P exams,Ā the weighting for each question is determined following the first administration an exam after is adopts new specification, at a "standard setting" or "cut score" meeting. That meeting is held in person. A new "cut score" meeting isn't needed for subsequent administrations that use the same specification.
I assume this meeting is just to determine new questions and answer choices, right?PAKS meetings determine the new exam specifications, and take place 1-2 years ahead of a new exam. It has nothing to do with cut scores.
This also makes me question that method, because not all the new questions in the test bank will be on the first exam. So, if the questionsĀ wereĀ weighted, they would have to sit for multiple administrations to get though review/weighting all the new questions?Generally speaking for P&P exams,Ā the weighting for each question is determined following the first administration an exam after is adopts new specification, at a "standard setting" or "cut score" meeting.
Correct. the standards are set once, and held for the rest of that specification.You are correct. However, the various committees don't set standards for future exams after each administration and, more importantly, they don't necessarily meet in person. It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results.
I don't know the answer to that. I suspect they work for NCEES onsite. I sort of doubt they have in-person meetings to go over a stray question here or there, too expensive; and I've never heard of it. If they do send it to external SMEs, those SMEs must be very very trusted. NCEES goes to great lengths to safeguard the questions so I doubt they want them leaving their control.So:
"At least two subject-matter experts, who are licensed engineers or surveyors, review the flagged items. In addition, the subject-matter experts consider examinee comments about specific exam questions. If the reviews confirm an error in a question, credit may be given for more than one answer."
These guys don't meet in person to review flagged questions?
These guys do meet in person?
Oh that brings up another question. On the front of my exam it said "all questions are weighted the same" which goes against everything I've been told.
Ah.But relative contribution of each question to the cut score is not equal. So as a general example one question may contribute 0.7 pts toward the cut score and another contributes 0.2 pts.
Oh that brings up another question. On the front of my exam it said "all questions are weighted the same" which goes against everything I've been told.
A different way to describe what RB said (more clear to me, maybe it will help you).Each correct answer is worth one point. An incorrect answer is worth zero points.
But relative contribution of each question to the cut score is not equal. So as a general example one question may contribute 0.7 pts toward the cut score and another contributes 0.2 pts.
You'll have to speak slowly and use small words with me, I'm not the sharpest crayon in the shed. lol
I'm having trouble keeping up with your posts, and doing my day job, so I'm a bit behind. I answered this above, but apparently after you wrote this."A new "cut score" meeting isn't needed for subsequent administrations" - Then how do they figure out the cut score if the questions are different (and according to ncees the questions are weighted the same)?
You can find the specifications for each exam on this page:"exam specification" - What do you mean by this? Like, are all the questions in the bank a "specification" or is morning session A + afternoon session C a "specificaiton"?
Who said that they are only taking the same exam that was offered a few weeks prior? Maybe they're only evaluating the he new questions too? Maybe they get a big enough sample size that they can use psychometrics for future new questions?This also makes me question that method, because not all the new questions in the test bank will be on the first exam. So, if the questionsĀ wereĀ weighted, they would have to sit for multiple administrations to get though review/weighting all the new questions?
No, this is to generate the new exam specifications. See webpage referenced above.I assume this meeting is just to determine new questions and answer choices, right?
Oh theĀ actualĀ exam specifications.You can find the specifications for each exam on this page:
Enter your email address to join: