The Collaborative International Dictionary of English v.0.48:

Crenation \Cre*na"tion\ (kr?-n?"sh?n), n.
   1. (Bot.) A rounded tooth on the edge of a leaf.
      [1913 Webster]

   2. The condition of being crenate.
      [1913 Webster]

WordNet (r) 3.0 (2006):

crenation
    n 1: one of a series of rounded projections (or the notches
         between them) formed by curves along an edge (as the edge
         of a leaf or piece of cloth or the margin of a shell or a
         shriveled red blood cell observed in a hypertonic solution
         etc.) [syn: scallop, crenation, crenature, crenel,
         crenelle]