Milner was the subject of a failed bid in May from City, who are expected to step up their pursuit of the England midfielder.
City boss Roberto Mancini has bolstered his options across the middle by buying David Silva and Yaya Toure, and Villa winger Downing suggests that if Milner joins them he may not necessarily become a regular in the first team.
"It will be interesting to see what XI (Mancini) starts with because he's got that much quality in that squad," Downing told The Guardian.
"They're all going to be fighting for places. It's going to be hard and it will be hard for James if he goes there.
"Will he play? I can't think how many midfielders they've got. I think that will be one of the decisions (that he has to consider): 'Do I want to go there and be a bit-part or stay here and play?'
"That's totally up to him. I am sure he's probably already made his decision."