Joe Royle was keen on landing the Australian goalkeeper, who went on to sign for Middlesbrough, but a deal was scuppered by the actions of Johnson, who was chairman until 1999 after initially arriving in 1994.
"Unfortunately at the time, it was a strange situation," Schwarzer told Sky Sports.
"Joe Royle was the manager and he was very keen on me coming to the club. Unfortunately the chairman of the club was playing, not even hardball, he wasn't the nicest.
"He took it as a bit of a game and I felt he wasn't interested in really signing me but I think Joe Royle pushed it so hard he felt he was obligated to bring me to the club and talk to me.
"Basically, after literally 10 minutes of discussion with the chairman we left the building and never returned."