This type of detailing is some of the most complicated there is... just the nature of the beast...
What I do is clean the non-skid first. Next tape off the non-skid next to any smooth gel-coat you want to compound, polish and wax. Buff the smooth gel-coat till finished. (whatever your process is), then remove the tape and apply the non-skid sealant.
The taping off is a paint but if you don't you'll get compound and wax residues into the textured non-skid. It's worth the time an energy in my opinion.
I can post pictures if you like?
