I would agree it sounds like a fundamentals thing. From what I can tell your looking to plug and chug the formula without the full understanding where the equation applies (or more important what the terms area).
The "N" terms are not the voltage. Its the number of turns for that coil. A true Auto transformer, for which this equation was derived, has a series coil "se" and a common coil "c" and they are "wired" in series. Really its just an adjustable center tap and the the ratio between the the two develops the equation referenced. So while I get what your doing, its just a terminology thing I'm point out.
While a single phase transformer can be connected like an Autotransformer to produce an electrically "equivalent" , its still physically a single phase transformer and is NOT a strait forward ratio between the primary and secondary. Draw out the single phase transformer circuits to visualize. The Single phase transformer wired as an auto-transformer has source voltage (480V) delivered to BOTH coils. The secondary coil (120V) actually acts against this supplied voltage to drop the voltage across the load to 360V.
So the easy answer after all is said and done (and I just came to this myself since I drew out the circuits in advance and established what needed to be done with each to avoid math for this on the test...) one of the terms must be considered negative. This is not an additive configuration. The auto-transformer equation assumes the coils are in series and additive, where-as that is not the case here and they counteract each other.
That is also why the "boost" version works. The equation assumes additive, boost is additive... so the result works.