Doyle told the club's website: "It gets to you after one game, two games, three and four games - and when it's five games it's a nightmare and it's not ideal.
"We played Swansea at home this week and we have to look to win that one, that's the only way we can look at it."
He added: "We probably deservedly got beaten, to be honest. We started well but they got their goal.
"In the second half we were a bit better but they got their second goal when we were just starting to get into the game. And goals change games.
"We just had two chances in the game really - other than that we huffed and puffed, but we didn't really look like scoring.
"Albion have got big, strong lads at the back and they made the game difficult."