The Brexit vote may put the brakes on what little forward momentum was left for British economic growth.