The song has long been rumoured to detail the breakdown of the couple's romance and the video even featured a girl bearing a striking similarity to Spears.
Timberlake has always denied the claims, but now his producer pal, Timbaland, has confirmed the Toxic hitmaker was the inspiration for the track as it was made following a heated phone conversation between the young lovers.
In a new E! True Hollywood Story special about Timbaland, the superproducer says: "He went to a concert and saw Britney, and Britney talked about him in the show and he was p**sed."
Timberlake, who stops short of mentioning Spears by name, explains: "I was on a phone call that was not the most enjoyable phone call. I walked into the studio and he (Timbaland) could tell I was visibly angry."
The singer poured out his feelings in the song, and the emotion showed in his opening lyrics.
Timbaland recalls: "I was like, 'Man, don't worry about it' and he was like, 'I can't believe she did that to me' and he was like, 'You were my sun, you were my earth...'"
The former 'N Sync star split from Spears that year. The track went on to land Timberlake the 2004 Grammy Award for Best Male Pop Vocal Performance.