Geralt's not great at helping himself. All he says is he was the only one who drew his sword against his attackers that day. Which is technically true. If they've a problem with a couple of dead robbers, so be it.