A post-Brexit trade deal with the US would not be enough on its own to make up for leaving the EU, a British minister said.