The Collaborative International Dictionary of English v.0.48:

Ordinarily \Or"di*na*ri*ly\, adv.
   According to established rules or settled method; as a rule;
   commonly; usually; in most cases; as, a winter more than
   ordinarily severe.
   [1913 Webster]

         Those who ordinarily pride themselves not a little upon
         their penetration.                       --I. Taylor.
   [1913 Webster]

WordNet (r) 3.0 (2006):

ordinarily
    adv 1: under normal conditions; "usually she was late" [syn:
           normally, usually, unremarkably, commonly,
           ordinarily] [ant: outstandingly, remarkably,
           unco, unusually]

Moby Thesaurus II by Grady Ward, 1.0:

60 Moby Thesaurus words for "ordinarily":
   again and again, all in all, all things considered, altogether,
   as a rule, as a whole, as an approximation, as per usual,
   as things go, as usual, at large, broadly, broadly speaking,
   by and large, chiefly, commonly, commonplacely, customarily,
   frequently, generally, generally speaking, habitually, in general,
   in many instances, mainly, many a time, many times,
   matter-of-factly, most often, mostly, naturally, normally,
   normatively, not infrequently, not seldom, oft, often,
   often enough, oftentimes, ofttimes, on balance, on the whole,
   overall, plainly, predominantly, prescriptively, prevailingly,
   prosaically, regularly, repeatedly, roughly, roughly speaking,
   routinely, simply, speaking generally, to be expected, typically,
   unseldom, usually, whenever you wish