3D analysis of skeletal volumes has become an important field in digital anthropology studies. The volume of the mastoid process has been proposed to display significant sexual dimorphism, but it has a complex shape and to date no study has quantified the full mastoid volume for sex estimation purposes. In this study we compared three different ways to isolate the volume of the mastoid process from digital 3D models of dry crania, and then evaluated the performance of the three different volume definitions for sex estimation purposes. A total of 170 crania (86 male, 84 females) excavated from five medieval Croatian sites were CT-scanned and used to produce 3D stereolitographic models. The three different isolation techniques were based on various anatomical landmarks and planes, as well as the anatomy of the mastoid process itself. Measurements of the three different mastoid volumes yielded different accuracies and precisions. Interestingly, anatomical structures were sometimes more useful than classical landmarks as demarcators of mastoid volume. For all three volume definitions, male mastoid volumes were significantly larger than female volumes, in both relative and absolute numbers. Sex estimation based on mastoid volume showed a slightly higher precision and better accuracy (71% correct classifications) than visual scoring techniques (67%) and linear distance measurements (69%) of the mastoid process. Sex estimation based on cranial size performed even better (78%), and multifactorial analysis (cranium size + mastoid volume) reached up to 81% accuracy. These results show that measurements of the mastoid volume represent a promising metric to be used in multifactorial approaches for sex estimation of human remains.