The likes of Matt Lowton and Christian Benteke are attracting interest from Premier League rivals.
Lambert said: "You just hope you keep building. If not, you have to go and get ones the same. If you're building something like that, you want to try and see it through.
"At the end of the day, the decision is my own if I let people go. That would always be the case. There's no point trying to plan something, thinking it's going to grow and then all of a sudden it's not there. You have to let it build."