You're not missing much. They're oftentimes too big and too full of useless items. They're for show.

[ Yes, castles have their uses but he'd still prefer to be out of them and on his ship. That's where he's comfortable. ]

I don't think you're missing all that much besides the fact that we have been, effectively, kidnapped. Not exactly sure why just yet.