Rescoring Test

[IlPadrino]

The thing is that you have diagnostic scores when you fail but when you pass you only get the congratulations. There is no way to know what did you do right. The diagnostic gives you percentages and unless you perform a McKeon and remember all the question on the test...How do you know?
The first time I failed did not come anywhere near 60%( based on the number of questions and the ones I think I got right by on the diagnostic). The second time I was very close to the arithmetic 70%. As a matter of fact it was hard to take for me. Again...that was based on the diagnostic.

Besides that...I did not say black magic. I said dark magic.....dark as :vadar:

Ohhh..... DARK magic says the DARK Knight. Got it!

You can work backwards from the diagnostic and the expected distribution of questions to get a very good indication of how many you got right.

The scores are normally distributed - it doesn't matter that we only have the left (fail) side of the curve.

 

[IlPadrino]

this cutting score is determined by the them after doing that exam and see the level of difficulty.

Do you have *ANY* evidence of this? I think you're wrong.

 

[Dark Knight]

Do you have *ANY* evidence of this? I think you're wrong.

You have good points Sray, but I think we are :deadhorse:

Here is the challenge, not for you, for anyone who passed. Can you tell with a 99% level of accuracy how many questions you had right? TMcK, if you are reading this help us out. I know you remember the 80 questions of your test. :17:

I remember a member who based on his diagnostic had 56 right answers and failed. He was/is a Civil Trasnpo. My second try I got 52 right answers...again...based on the diagnostic. :Failed: that time.

Seriously...I do not think anyone outside NCEES can explain the logic of the cut score. As I said...it is dark magic. :vadar:

 

[Mike in Gastonia]

Do you have *ANY* evidence of this? I think you're wrong.

From the ncees website:

The goal of equating is to ensure that an examinee's chances of passing remain constant regardless of the particular administration of the exam that was taken. Put another way, this means that an examinee is not penalized if the exam taken is more difficult than usual. If the exam is more difficult than usual, fewer questions must be answered correctly to achieve a passing score.

I read this to mean that do an analysis to compare this exam's takers' intelligence to the previous ones and then compare how they did on their respective exams. If the groups are about the same, intelligence wise, and the second group did worse on their exam, they lower the pass score. Or if the second group did better, they raise it.

 

[RIP - VTEnviro]

Thanks as always for the sagely wisdom MIG.

 

[Jennifer]

If it helps...Last time, I figured my cut score from my diagonastic report to be around 58% and my score was a 66%...I think this probably varies by discipline and exam administration, though. That was for structural 1

 

[Vishal]

When I said that 70% is not equal to passing and is not equal to 56 questions out of 80, I mean that in your backwards calculation theory you are "assuming" that all the questions have been considered to achieve the final score. This is too much to assume. Your 70% cut score theory (even with considering 'significant difference') is still based on all the questions being taken into consideration.

How do you know that from which section how many questions were discounted? In Civil AM, you have 8 questions on 5 subjects. Say only 4 questions from Construction were accounted in the final score and in the diagnostic you are told that you scored 50%. So by your calculation you got 4 questions correct whereas in "reality" it is only 2. There you go... the significant difference you defined as 2 questions.

Please tell me if I am wrong, or you have some other ideas.

Also, not to forget that there are other disciplines that may have different cut scores and different distribution of questions. It is just too variable to be able to come to a definitive answer or even in a range.

 

[Mike in Gastonia]

When I said that 70% is not equal to passing and is not equal to 56 questions out of 80, I mean that in your backwards calculation theory you are "assuming" that all the questions have been considered to achieve the final score. This is too much to assume. Your 70% cut score theory (even with considering 'significant difference') is still based on all the questions being taken into consideration.
How do you know that from which section how many questions were discounted? In Civil AM, you have 8 questions on 5 subjects. Say only 4 questions from Construction were accounted in the final score and in the diagnostic you are told that you scored 50%. So by your calculation you got 4 questions correct whereas in "reality" it is only 2. There you go... the significant difference you defined as 2 questions.

Please tell me if I am wrong, or you have some other ideas.

Also, not to forget that there are other disciplines that may have different cut scores and different distribution of questions. It is just too variable to be able to come to a definitive answer or even in a range.

I think the "discounting" of questions is one of those urban legends. Why would they discount it? they say they don't have a "curve". If they go through and analyze them and find that there is an error and two answers could be considered correct, wouldn't they just give credit to everyone who selected one of those two answers? I guess if they had one that was just wrong, they would give everyone credit, which I guess is the same as throwing it out.

 

[Vishal]

I think the "discounting" of questions is one of those urban legends. Why would they discount it? they say they don't have a "curve". If they go through and analyze them and find that there is an error and two answers could be considered correct, wouldn't they just give credit to everyone who selected one of those two answers? I guess if they had one that was just wrong, they would give everyone credit, which I guess is the same as throwing it out.

The discounting of the questions happens when there is insufficient data or ambiguity involved or if there is some out of whack question. If there are two possible answers then I would rather have them discount the question than give the credit to everyone since in giving the credit to everyone those who guessed on that question will benefit where in that is not desirable for the integrity of the test and also for the public safety.

Just fellt like using heavy words.... but seriously I would still have the questions taken out of consideration than giving mass credit.

 

[Jennifer]

http://www.ncees.org/exams/scoring/scoring_method.php

 

[mudpuppy]

The scores are normally distributed - it doesn't matter that we only have the left (fail) side of the curve.

That's an interesting assumption. How do you know the scores are normally distributed?

 

[IlPadrino]

That's an interesting assumption. How do you know the scores are normally distributed?

The Law of Large Numbers and the Central Limit Theorem require them to be...

 

[IlPadrino]

From the ncees website:


I read this to mean that do an analysis to compare this exam's takers' intelligence to the previous ones and then compare how they did on their respective exams. If the groups are about the same, intelligence wise, and the second group did worse on their exam, they lower the pass score. Or if the second group did better, they raise it.

And also from the same website:

Are the exams graded on a curve? Is there a target pass rate?No. NCEES scores each exam based on its own merits with no regard for a predetermined percentage of examinees that should pass or fail. All exams are scored the same way.

So how does this balance with

For subsequent administrations, the same passing standard is preserved using a statistical technique known as equating. This technique is widely used by testing experts for occupational licensing exams. The goal of equating is to ensure that an examinee's chances of passing remain constant regardless of the particular administration of the exam that was taken. Put another way, this means that an examinee is not penalized if the exam taken is more difficult than usual. If the exam is more difficult than usual, fewer questions must be answered correctly to achieve a passing score.

Also,

Using a process that is widely accepted in the testing profession, the committee assesses the examination performance that may be expected of a minimally competent examinee. After the NCEES examination committee reviews the committee's work for reasonableness, the passing score is established.

is an interesting passage.

 

[IlPadrino]

I was reading up on equating and found a straight-forward explanation and an interesting detail of one method.

On second thought, I think my quote "Do you have *ANY* evidence of this? I think you're wrong." was *WRONG* (I apologize to noleengineer!), but I still believe with enough fail diagnostics, a cut score can be estimated within a point or two.