The Dons currently sit ninth on the Scottish Premier League table but Brown feels he can still get the best out of his squad amidst murmurings of him losing his job at Pittodrie.
"It is up to the board if I am given the time to get success," said Brown who took the reins in 2010.
"But it would be unfair if I wasn't because it takes time and this is a work in progress - and it's very nearly there.
"I am very optimistic we can still bring success to Aberdeen. It's not all doom and gloom.
"A few supporters are disgruntled but hopefully the fair-minded ones can see what is happening and understand.
"But it's an over-simplification to say change the manager. The fans and directors may say that but that is up to them.
"Changing a manager is not a magic wand. I love this football club and if they don't think this is for me I'd be disappointed."