Many Americans are left off their Olympic team because they just barely miss making the top 3.

One of your top-3 have more chances to get the Gold medal your unlucky top-4 will not get, and there's only one Gold per event.

The US is sending about 500 athletes to get about 100 medals.
They could send 1000 and not get one single more medal.

Besides, of split into 50 teams, the US would lose many team and relay medals.

One balances the other.
Arithmetic addition is not perfect but is far from incongruous.
It's the only simple way to do, anyway.