Both Yowie and Nolem offer good advice!
Swell wraps around headlands depending on the direction its coming from and will break on any ledge on an angle. The intensity drops as it wraps, but it will still impact the ledge.
So your Easterly ledge will still get swell from ALL directions.
I am in agreement with nolem, in that I am not a believer in 'freak waves'. Swell sets may/do join each other especially when swells are coming from multiple directions resulting in a larger set.
There may also be a wind swell on top of a base swell directions.
When rockfishing you need to be watching the swell way out to sea . . eg 200, 300, 400m not just at the waves about to hit your ledge. That way you can see larger waves in time to make a move.
If I was you I'd be starting off with a guide such as Alex B, he will give you a good starting point. Most importantly, it will happen safely!