Buyout rumors always swirl around Avago, and for good reason. However, Broadcom might prove to be a tough bite to swallow.