England and Chelsea defender Terry, 31, is accused of calling Ferdinand a "f****** black c***" during a Premier League match last October.
On Monday, QPR player Ferdinand told Westminster Magistrates' Court that initially he did not think any racist terms had been used.
But after the match his girlfriend at the time played him a YouTube clip, and he became convinced that one had.
Ferdinand told the court that if he had realised at the time he would have told officials.
He said: "I would have been obviously very hurt and I probably wouldn't have reacted at the time because, being a professional, you can't do that. I probably would have let the officials know what happened and dealt with it after the game.
"When someone brings your colour into it, it takes it to another level and it's very hurtful."
Chelsea were down to nine men in the clash at Loftus Road on October 23, and Ferdinand and Terry began trading insults over a penalty claim, the court heard.
Ferdinand said: "He called me a c*** and I called him a c*** back and he gave me a gesture as if to say my breath smelled.
"I said to him 'How can you call me a c***? You sh***** your team-mate's missus, you're a c***'."
This was a reference to Terry's alleged affair with Wayne Bridge's ex-girlfriend, Vanessa Perroncel.
Ferdinand jogged down the pitch making a fist gesture to imply sex, he told the court.
Terry then met Ferdinand to ask what had happened.
"Mr Terry said 'Do you think I racially abused you?' I was like 'No'," Ferdinand told the court.
"I said 'No, that never came out of my mouth'. Then Ashley Cole popped his head round and said 'Yeah, didn't you say that to me?' I said 'I didn't say that at all'."
He agreed it was just "handbags", and "banter", the court heard.
However, when he later viewed film footage of the incident, he believed a racist term had been used.
Terry is accused of a racially aggravated public order offence, which he denies. He maintains that he was only sarcastically repeating what Ferdinand wrongly thought he had said.