In order to investigate the relationship between the occurrence of the “grubby” sensory defect caused by olive fruit fly (Bactrocera oleae (Rossi)) infestation and the resulting volatile composition, virgin olive oils were extracted from olives of the Leccino cultivar with 0%, 50%, and 100% olive fly infestations and subjected to analysis of the basic chemical quality parameters, fatty acids and volatiles, and sensory analysis by the Panel test. A 100% olive fly infestation reduced the basic chemical quality of the oil, while the fatty acid composition was not affected in any case. The overall sensory quality score and intensity of the positive sensory attributes decreased, while the intensity of the “grubby” defect increased proportionally to the degree of infestation. The occurrence and intensity of this defect were clearly causally related to the concentrations of 3-methylbutanal, 2-methylbutanal, β-ocimene, ethyl 2-methylbutyrate, dimethyl sulfoxide, 4-methyl-5H-furan-2-one, α-farnesene, 6-methyl-5-hepten-2-one, 1-octanol, E-2-nonen-1-ol, benzeneacetaldehyde, heptanal, and octanal, implying that the perception of “grubby“ comes from their joint contribution to the overall olive oil flavour. In addition to contributing to the understanding of the chemical origin of “grubby”, the results obtained could potentially be used to develop strategies to support sensory analysis in the classification of olive oil quality and the confirmation of the presence of this sensory defect in oil samples.