one example would be to do a channel blur on a layers alpha (or fast blur over the entire layer, etc)... but then CC composite and set it to stencil alpha.
now the edge blur gets contained in the layers original alpha
this is the alpha channel ...the top here is just a channel blur, the bottom has the CC composite as well