October 2019 Post Exam Wait Period - Welcome to the Suck

[Will.I.Am PE]

Sounds like you're channeling your inner @RBHeadge PE

Most definitely.

 

[Orchid PE]

The Control Systems cut score meeting.

The PE control systems exam has a new specification for this exam administration. When that happens, the subject matter experts meet in person to determine the cut score and set standards for future exams. This is usually one of the last things that takes place in the exam scoring process. Results typically follow within a week after that. This year, the Thanksgiving holiday is the last week in November. This will, very likely, push the control systems cut score meeting to the first week (or so) in December, which will, in turn, probably push the results to the second week in December.

According to NCEES, "After each pencil-and-paper exam administration, every exam undergoes extensive statistical analysis to determine the minimum level of performance required for entry into the profession. This rigorous process helps ensure an equitable licensure system for examinees while upholding the standards of the profession."

To me this reads that a cut-score is determined for every each administration, not just exams with new specifications, but I could be wrong.

Step 4 of their process: "When the analyses and reviews are completed, NCEES changes the answer keys as necessary. The passing score and final correct answers for each exam are then used to score all answer sheets. A percentage of the answer sheets are manually verified, and the results are compared to the machine score to ensure accuracy."

Again, this leads me to believe that the cut-score has to be determined after each administration.

Maybe someone could answer me this, I know that there are multiple exam variations for different exam sessions. For example, there are multiple morning session exams (like morning exam 1, 2, 3, etc.) and multiple afternoon session exams. Does anyone know if there are a finite number of exams, or is there a question bank that questions are pulled from to generate an exam session? Some things that lead me to believe the latter is true is a) redacted, and b) there was a duplicate question on the power afternoon session. This last part makes me wonder if the exams are automatically generated from a question bank, administered, and then the committee sets the cut-score.

Maybe someone who has participated in PAKS could enlighten me.

 

[Will.I.Am PE]

To me this reads that a cut-score is determined for every administration, not just exams with new specifications.

You are correct. However, the various committees (or whatever they are) don't set standards for future exams after each administration and, more importantly, they don't necessarily meet in person. It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results. Correct me if I'm wrong, @RBHeadge PE.

Either way, if you'd like something more definitive than just my (and RB's) word, I can send it to you. I'm new at reading the tea leaves, but it's been very consistent over the last 10, or so, years. Just let me know.

 

[Orchid PE]

You are correct. However, the various committees don't set standards for future exams after each administration and, more importantly, they don't necessarily meet in person. It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results. Correct me if I'm wrong, @RBHeadge PE.

Either way, if you'd like something more definitive than just my (and RB's) word, I can send it to you. I'm new at reading the tea leaves, but it's been very consistent over the last 10, or so, years. Just let me know.

Yeah send it to me. I'm bored and just interested in their process. I updated my post, btw, with some more information/questions.

 

[Orchid PE]

However, the various committees don't set standards for future exams after each administration

Typo on my part. Edited.

 

[Will.I.Am PE]

Yeah send it to me. I'm bored and just interested in their process. I updated my post, btw, with some more information/questions.

My employer blocks EB on my work computer, so I may or may not end up sending it this afternoon on my phone. If all else fails, you'll get it this evening, after I get home.

 

[Orchid PE]

It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results.

So:

"At least two subject-matter experts, who are licensed engineers or surveyors, review the flagged items. In addition, the subject-matter experts consider examinee comments about specific exam questions. If the reviews confirm an error in a question, credit may be given for more than one answer."

These guys don't meet in person to review flagged questions?

The PE control systems exam has a new specification for this exam administration. When that happens, the subject matter experts meet in person to determine the cut score and set standards for future exams.

These guys do meet in person?

Doesn't make too much sense to me since the questions change from one administration to the next. So this past Oct Controls exam will be different from this coming Aprils. And if they had to meet in person to go over the questions on the October exam, why wouldn't they need to meet in person to review April's exam questions since they are different from October's?

It makes me think that the "at least two subject-matter experts" would meet in person to review each exam administration since they are all different. I would understand them not needing to meet in person if each exam was the same. Then sure, they only need to review the first one. But since each exam session of each administration has different questions, I would think it's like reviewing a whole new exam each time.

 

[Tiger15]

Yep, the exam changes with every administration. There are some people on here who have taken it multiple times. They could probably share more detail (without revealing any questions!)

So a cut score has to be determine for each exam for each administration.

Now I do think the exam are the same within each administration. Like all Oct '19 Power exams are exactly the same. Something you said made that seem unclear to me. I'm also wondering how that will change with the transition to CBT.

 

[Orchid PE]

Now I do think the exam are the same within each administration. Like all Oct '19 Power exams are exactly the same. Something you said made that seem unclear to me. I'm also wondering how that will change with the transition to CBT.

I agree. I think they're the same at each administration. I could say I know that for a fact, but I won't say it.

 

[RBHeadge PE]

According to NCEES, "After each pencil-and-paper exam administration, every exam undergoes extensive statistical analysis to determine the minimum level of performance required for entry into the profession. This rigorous process helps ensure an equitable licensure system for examinees while upholding the standards of the profession."

To me this reads that a cut-score is determined for every administration, not just exams with new specifications, but I could be wrong.

Step 4 of their process: "When the analyses and reviews are completed, NCEES changes the answer keys as necessary. The passing score and final correct answers for each exam are then used to score all answer sheets. A percentage of the answer sheets are manually verified, and the results are compared to the machine score to ensure accuracy."

Again, this leads me to believe that the cut-score has to be determined after each administration.

Yes @Chattaneer , each administration of every exam has a different cut score. But the cut score is determined based on the component questions. Generally speaking for P&P exams,  the weighting for each question is determined following the first administration an exam after is adopts new specification, at a "standard setting" or "cut score" meeting. That meeting is held in person. A new "cut score" meeting isn't needed for subsequent administrations that use the same specification.

Maybe someone could answer me this, I know that there are multiple exam variations for different exam sessions. For example, there are multiple morning session exams (like morning exam 1, 2, 3, etc.) and multiple afternoon session exams. Does anyone know if there are a finite number of exams, or is there a question bank that questions are pulled from to generate an exam session?

The questions come from a question bank. I wouldn't ever expect to see the same exam (questions and order) offered twice.

Maybe someone who has participated in PAKS could enlighten me.

PAKS meetings determine the new exam specifications, and take place 1-2 years ahead of a new exam. It has nothing to do with cut scores.

 

[Orchid PE]

Generally speaking for P&P exams,  the weighting for each question is determined following the first administration

Oh that brings up another question. On the front of my exam it said "all questions are weighted the same" which goes against everything I've been told.

 

[Orchid PE]

Yes @Chattaneer , each administration of every exam has a different cut score. But the cut score is determined based on the component questions. Generally speaking for P&P exams,  the weighting for each question is determined following the first administration an exam after is adopts new specification, at a "standard setting" or "cut score" meeting. That meeting is held in person. A new "cut score" meeting isn't needed for subsequent administrations that use the same specification.

You'll have to speak slowly and use small words with me, I'm not the sharpest crayon in the shed. lol

"each administration of every exam has a different cut score"  - Makes sense.
"A new "cut score" meeting isn't needed for subsequent administrations" - Then how do they figure out the cut score if the questions are different (and according to ncees the questions are weighted the same)?
"exam specification" - What do you mean by this? Like, are all the questions in the bank a "specification" or is morning session A + afternoon session C a "specificaiton"?

 

[Orchid PE]

PAKS meetings determine the new exam specifications, and take place 1-2 years ahead of a new exam. It has nothing to do with cut scores.

I assume this meeting is just to determine new questions and answer choices, right?

 

[Orchid PE]

Generally speaking for P&P exams,  the weighting for each question is determined following the first administration an exam after is adopts new specification, at a "standard setting" or "cut score" meeting.

This also makes me question that method, because not all the new questions in the test bank will be on the first exam. So, if the questions were weighted, they would have to sit for multiple administrations to get though review/weighting all the new questions?

 

[RBHeadge PE]

You are correct. However, the various committees don't set standards for future exams after each administration and, more importantly, they don't necessarily meet in person. It's scheduling the in-person meeting (which is typically one of the final steps in the scoring process) that typically delays results.

Correct. the standards are set once, and held for the rest of that specification.

So:

"At least two subject-matter experts, who are licensed engineers or surveyors, review the flagged items. In addition, the subject-matter experts consider examinee comments about specific exam questions. If the reviews confirm an error in a question, credit may be given for more than one answer."

These guys don't meet in person to review flagged questions?

These guys do meet in person?

I don't know the answer to that. I suspect they work for NCEES onsite. I sort of doubt they have in-person meetings to go over a stray question here or there, too expensive; and I've never heard of it. If they do send it to external SMEs, those SMEs must be very very trusted. NCEES goes to great lengths to safeguard the questions so I doubt they want them leaving their control.

Oh that brings up another question. On the front of my exam it said "all questions are weighted the same" which goes against everything I've been told.

Each correct answer is worth one point. An incorrect answer is worth zero points.

But relative contribution of each question to the cut score is not equal. So as a general example one question may contribute 0.7 pts toward the cut score and another contributes 0.2 pts.

 

[Orchid PE]

But relative contribution of each question to the cut score is not equal. So as a general example one question may contribute 0.7 pts toward the cut score and another contributes 0.2 pts.

Ah.

 

[Tiger15]

Oh that brings up another question. On the front of my exam it said "all questions are weighted the same" which goes against everything I've been told.

Each correct answer is worth one point. An incorrect answer is worth zero points.

But relative contribution of each question to the cut score is not equal. So as a general example one question may contribute 0.7 pts toward the cut score and another contributes 0.2 pts.

A different way to describe what RB said (more clear to me, maybe it will help you).

The difficulty of the question is weighted and considered when determining the cut score. Let's say hypothetically they determine the cut score is 68 (out of 80) based on the difficulty and expectations for each question. Once that cut score is determined, you have to get 68 right to pass. It doesn't matter if your 68 points come from easy questions or hard questions. They all count the same toward your pass/fail.

 

[Orchid PE]

@RBHeadge PE, @jean15paul_PE So like this for an example session (but normally done for the entire exam):


Question #


Right/Wrong


Cut Weight


1


1


0.2


2


1


1.0


3


1


0.4


4


0


0.1


5


1


0.6


6


0


0.6


7


1


0.2


8


0


0.4


9


0


0.9


10


1


0.7


11


1


0.9


12


1


0.2


13


0


0.6


14


1


0.6


15


1


1.0


16


1


0.1


17


1


0.6


18


0


0.5


19


1


0.7


20


0


0.6


21


1


0.8


22


1


0.9


23


1


0.3


24


1


0.5


25


1


0.9


26


0


0.4


27


1


0.7


28


1


0.7


29


1


1.0


30


0


0.3


31


1


0.4


32


0


0.5


33


1


0.6


34


1


0.2


35


1


0.3


36


1


0.8


37


1


0.2


38


1


0.5


39


1


0.6


40


0


0.4


 


29


21.9


 


Exam Score


Cut Score

 

[RBHeadge PE]

You'll have to speak slowly and use small words with me, I'm not the sharpest crayon in the shed. lol

gif posted to break the tension

"A new "cut score" meeting isn't needed for subsequent administrations" - Then how do they figure out the cut score if the questions are different (and according to ncees the questions are weighted the same)?

I'm having trouble keeping up with your posts, and doing my day job, so I'm a bit behind. I answered this above, but apparently after you wrote this.

"exam specification" - What do you mean by this? Like, are all the questions in the bank a "specification" or is morning session A + afternoon session C a "specificaiton"?

You can find the specifications for each exam on this page:

https://ncees.org/engineering/pe/

This also makes me question that method, because not all the new questions in the test bank will be on the first exam. So, if the questions were weighted, they would have to sit for multiple administrations to get though review/weighting all the new questions?

Who said that they are only taking the same exam that was offered a few weeks prior? Maybe they're only evaluating the he new questions too? Maybe they get a big enough sample size that they can use psychometrics for future new questions?

I assume this meeting is just to determine new questions and answer choices, right?

No, this is to generate the new exam specifications. See webpage referenced above.

 

[Orchid PE]

You can find the specifications for each exam on this page:

Oh the actual exam specifications.