Just something I thought about randomly, during my musings about this test....
How does NCEES make the test difficulty consistent? What I mean by this is, I place myself in the shoes of someone writing a test for practicing engineers...many with PEs, many with Master's degrees. However, I cannot predict the quality and preparation level of the prospective examinees. Nor can I accurately predict which subjects will be particularly difficult to that examinee group.
The only real assumptions I can make are that everything in the NCEES and PPI practice exams will be "easy" for most examinees. Other than that, I don't have a way to foresee it. Hence...why I'm intrigued by the consistent pass rate, especially for repeat test takers. Imagine a scenario where NCEES writes a test that involves for problems in the afternoon, and they are extremely simple, consisting of determining "Cs" for four buildings...and that's it. Would the pass rate still be 30% +/- 3% as it has been nearly without exception for the past decade?
I am curious if the cut score is determined immediately after the test is written, or during grading, or after grading. Unless the exam is actually tested before hand on some representative group of individuals, it is impossible to determine how many people will find the test difficult and how many will pass. So although NCEES swears up and down that they do not adjust the cut score to prevent 70% of people from passing....I don't understand how they can hit that passing rate so consistently without much, if any, variation.
Another thing...NCEES has been very successful at keeping the grading criteria secret for the afternoon problems. The US government has had leakers throughout its history, but they have a thing or two they can learn from NCEES. The secrecy of the grading criteria coupled with the consistently low pass rate leads me to an interesting conclusion...there must be fluctuation in how essay questions are scored, and part of that fluctuation might be based on how an examinee did in comparison to other individuals. Otherwise you would have tests which are too difficult and the pass rate would be very low. Or the exam would be based on things everyone was familiar with...and the pass rate would be 50% or higher.