Downing has refused to commit to a new contract at Villa Park.
He said: "You hope they do (understand) but normal fans will want you to stay at their club for the rest of your life. If you do leave there's obviously a bit of bitterness there.
"You hope they do understand because, it's like any job, if there's a better job on hand, or a better level to do it, then people do it.
"If I don't move and I stay, I'll get on with the job as normal. I'll do my best and give 100%.
"It's up to their fans and their opinions of me. If they want to be loud and clear with what they think of me, then that's fine.
"I'll still give my best because I'm employed to do so."