That's correct. USB is host driven architecture. Xfers actually need CPU, and the more devices you have, the less each individual device gets. From memory, I'm not sure if USB supports priority for devices, but I'd guess mice would get rather low bandwidth.
Now, Firewire, on the other hand, is driven by hardware... and you're supposed to get more consistent throughput as well, even thought the size of the pipe is slightly narrower than USB2.