Allegri was speaking before yesterday's defeat to Fiorentina.
He told Gazzetta.it: "The presence of Guardiola's entourage in the crowd didn't worry me, there have been countless names and rumours circulating.
"The story has bored me a bit and it's up to the club to deal with these situations. Milan has different problems - recovering in the league and qualifying from the Champions League.
"It's up to me to achieve these objectives and I will do my best for as long as I'm on this bench."