On a previous visit I was told that this is a "folly" but today I realised that is [was] a shell house. It would appear that it was decorated with shells inside rather than outside.

A shell is usually a small building, of distinctive shape, and decorated with sea shells. The degree of decoration can vary, from simple lines or features, to a full wall and ceiling covering. Time consuming to make, expensive now to maintain and restore, they are always going to be a rarity.