When I was a kid back in the seventies, most semi trucks seemed to be pulled cab over engine (COE) and conventional tractors in almost equal numbers. Now, I can't tell you the last time I saw a COE tractor. Was there some of economic/legislative/tech reason that the COE became extinct, or at least endangered, in America?
The law is what made them popular to begin with. Prior to 1956, the maximum length for a vehicle was 42 feet. The Federal Aid Highway Act of 1956 gave us interstates and 65 feet long trucks. Legislation in 1976 moved that to a max of 75 feet.
With less restrictions on length, you can see the obvious move to the current design (less engine heat in the cab, less noise, more "crumple" zones, aerodynamics, etc). That being said, the COE design is still popular in other countries where length is still restricted (notably Japan).