The reputation system makes no sense sometimes

I just did the weapons shipment mission for the Ashiga on Tatooine. Why tf do I lose reputation with the Hutts if I don't double cross the Ashiga? The Hutts have no idea I'm involved at all.

There's other missions like this too. The trade off should be higher rewards for the double cross vs loss of rep with the employer. Losing rep with the other faction makes no sense unless you are seen during the mission