Lords also warned against plans to take the UK out of the customs union