Castles have a better defensive strength than cities and forts. They have stronger walls and their centers are next to the main building which attacks all enemies at a very long range. That makes the castle even harder to conquer.
They get a better defense, but they lose a notable amount of tax income till the maximum level. They also can't recruit diplomats, merchants, spies and assassins till the maximum level. They have one more recruitment slot than the cities have.