The striker, who has joined Caley from Coleraine, says he turned down the opportunity to take on a Premier League challenge with the Swans.
Allen told the Daily Record: "I had a couple of other options. Swansea showed an interest but once I spoke to Inverness I was really keen to go there and it seemed the perfect move.
"I'm relishing the move. It's a big step up from the Irish League but I will go there full of confidence."