Possibly dried out finger tips for me, but that's about it.
Possibly it's the magnesia in the Choseras that doesn't like your skin?
That's the main difference of Choseras to anything else, and as far as I'm aware most of the stuff in man made stones is pretty much inert, biologically speaking. Just abrasive and a binder, and the abrasive shouldn't cause much problem. A quick google says that magnesia is actually a good thing for skin problems a lot of the time...
Most likely the residue being hygroscopic and sucking the moisture from your skin. Wash hands properly with soap and if possible warn/hot water after using the stones and maybe a good skin cream if needed?
(I'm not one to use skin cream, but the pretty young lady who lives here does...)
Good luck, and if you find out anything specific do please let us all know.
Stu.