No matter how powerful he was, they were not willing to die for him. So after they heard his decision to continue besieging the city, many of them didn't move to take the city. They stayed where they were with a conflicted look on their faces as they wondered if Loki would try to kill them if they disobeyed him.
When Loki saw that they were hesitant, he snorted and said, "You are free to go if you don't want to stay."
Then he started walking towards the city to lay claim to it. Some people decided to follow him. But more than half of them chose to return the way they came, and they ran away as fast as possible.
Loki didn't like it, but he couldn't blame them. This was because it was a safe idea for them to give up now.
Even if he blamed them, he couldn't stop all of them and force them to help him hold the city. They would just run away if he used force on them. Besides, he wasn't their leader, so he had no right to order them around.
