He only took over from David Moyes at the Stadium of Light in June following the club's relegation from the Premier League, but he lasted just 15 games.
Grayson had been on a three-year contract.
"The club would like to place on record its thanks to Simon and his staff for their efforts during their time at the football club.
"Chief executive Martin Bain said: “Simon and his team have worked tirelessly to achieve the best for the football club during their time here.
“While we hoped that Simon's experience in the Football League would help us to a successful season, results have not been good enough for a club of this stature.
“In order for us to improve upon our current position we believe a fundamental change is necessary.
"Assistant manager Glynn Snodin also departs."