Fluorite single crystals, CaF2 are multifunctional crystals that have potential applications in environmental, thermo-luminescent and accidental dosimetry. A comprehensive investigation of the co-located naturally occurring colourless and coloured (yellow, sea-green, purple, blue, black) fluorites (CaF2) doped with bivalent and multivalent cations was carried out for the cationic and anionic sub-lattice defects generated due to natural irradiation from associated actinide containing pyrochlore grains. Detailed spectroscopic analyses have been used to map the various cationic and anionic defect structures such as O₃^– and O^2––Vacancy type defect complexes in yellow fluorites, F-centres and irradiation induced metallic Ca in purple, blue and black fluorites. The confinement of the radiation damage only to the anionic sub-lattice was explained in terms of higher displacement cross-section for F (59 b) as compared to Ca (33 b) as well as lower energy requirement to create F displacements and vacancies. It is observed that the fluorite retained its phase stability under natural radiation exposure and elemental substitutions.