Blackburn have now gone eight consecutive League games at West Ham without a win as they slumped to a 2-1 defeat at Upton Park.
They even had the perfect start when leading scorer Roque Santa Cruz gave them the lead with his 16th strike of the campaign in the 19th minute - a diving header from a Morten Gamst Pedersen cross.
But just before the break Dean Ashton pulled the Hammers level and for the second Saturday Blackburn conceded a late goal as youngster Freddie Sears headed the winner to end West Hams run of three successive defeats.
Pedersen could have regained the lead for the visitors in the second half but, unmarked, fired over from eight yards.
For Blackburn the season is in danger of running out of steam after a strong start and any hopes of automatic qualification for Europe next year are rapidly disappearing.
"We seem to have a problem, for whatever reason, whenever we come here," Hughes told Sky Sports News.
"West Ham seem to be our bogey team and it has happened again. We can't quite put our finger on the reason why that seems to happen.
"But probably today it is a situation that we didn't quite have enough drive in our play when we needed to ask a lot more questions of West Ham.
"In the first half there was a little bit of apprehension about the place and we should have capitalised on that."
Hughes has admitted in the past that his side may have to settle for a place in the Intertoto Cup this season, which would require a sixth-placed finish.
"We have still got games to play and games that we feel we can win," added Hughes.
"Maybe the damage hasn't been as key as possibly it could have been, but we will have to wait and see.
"It is only at the end of the season when we will see if this defeat has a real impact on us."