Dortmund had it all to do after losing the first leg 3-0 in Spain but last season's runners-up looked on course to stage a remarkable comeback when, after seeing Madrid winger Angel di María miss an early penalty, they scored twice through Marco Reus during a first-half onslaught.
In the 64th minute the visitors survived a huge let off when Henrikh Mkhitaryan missed an open goal.
The Armenia international looked to have done the hard work as he raced onto Reus' through-ball and rounded Casillas but, with the goal at his mercy, he shot against the near post.
Dortmund had found their second wind though and in the next six minutes Casillas was called on to make to crucial saves to deny Mkhitaryan and Kevin Großkreutz.
The German side continued to throw everything at Madrid in the second half, but they could not find the third goal they needed as Carlo Ancelotti's men held on for a 3-2 aggregate win.