The cost is lost opportunity, but under most circumstances that opportunity carries more downside than others, and should be avoided.
At my level of trading, many swaps are invisible: One can go for days without seeing a tariff, or an advantage.
Most of the time the windows I color black and get aggressive in do not go against the swap.