You using the same rig and bait?
I find that most guys that lose their baits when fishing drummer off the rocks are doing three things
1. Not fishing deep enough if using a float
2. Not fishing heavy enough. As the baits drifts down the water column, the longer it takes to get down to the strike zone the more likely the bait will not make it intact.
3. Bait size. Larger baits catch larger fish... Larger baits bring all sorts of fish. Small fish will peck at it and it gives the bait time for a bigger fish to come have a nip