I know this must sound very stupid. But is there a rule, some nugget passed down by sages that tells the apprentice when to genuinely normal map, or when to just use Crazybump/NVidia Filter?

Inversely what things should be left to geometry as opposed to represented by texturing?

forgive my ignorance in advance, I have had no one explain such things.

The hows, and the whys are not always synonymous.

Small surface detail.. basicly what would be decribed as the "texture" of a material by a non 3D artist can be nicely done with a 2D filter. Most of the time this applies to enviromental art as in playermodels this small detail is almost always merged with bigger detail.
It is possible however to merge normal maps with several programms (or by hand which is more difficult) and thus the small detail can be made with a filter and the bigger with a mesh.

About the geometry VS texture question: Normal maps can very convincingly be used for concave detail which is not part of the siluette of the mesh (convex detail most of the time is, but if it is not a normal map can be used for that too). Everything else should be made with real geometry (but can be "smoothed" in its lightning information by normal maps).

Hope that helps.