The smell of dirt is actually from a molecule called 'geosmin' that is produced by a strain of bacteria. ... It is also possible that you are mistaking an iron rich smell with dirt smell - in which case it could be something quite serious (if your stool color is black or tarry - definitely see a doctor).
