The Securities and Exchange Commission is about to levy what could be the largest fine ever against a market operator.