Objectives: Prior studies relating body mass index (BMI) to brain volumes suggest an overall inverse association. However, BMI might not be an ideal marker, as it disregards different fat compartments, which carry different metabolic risks. Therefore, we analyzed MR-based fat depots and their association with gray matter (GM) volumes of brain structures, which show volumetric changes in neurodegenerative diseases. Methods: Warp-based automated brain segmentation of 3D FLAIR sequences was obtained in a population-based study cohort. Associations of temporal lobe, cingulate gyrus, and hippocampus GM volume with BMI and MR-based quantification of visceral adipose tissue (VAT), as well as hepatic and pancreatic proton density fat fraction (PDFFhepatic and PDFFpanc, respectively), were assessed by linear regression. Results: In a sample of 152 women (age 56.2 ± 9.0 years) and 199 men (age 56.1 ± 9.1 years), we observed a significant inverse association of PDFFhepatic and cingulate gyrus volume (p < 0.05) as well as of PDFFhepatic and hippocampus volume (p < 0.05), when adjusting for age and sex. This inverse association was further enhanced for cingulate gyrus volume after additionally adjusting for hypertension, smoking, BMI, LDL, and total cholesterol (p < 0.01) and also alcohol (p < 0.01). No significant association was observed between PDFFhepatic and temporal lobe and between temporal lobe, cingulate gyrus, or hippocampus volume and BMI, VAT, and PDFFpanc. Conclusions: We observed a significant inverse, independent association of cingulate gyrus and hippocampus GM volume with hepatic fat, but not with other obesity measures. Increased hepatic fat could therefore serve as a marker of high-risk fat distribution. Key Points: • Obesity is associated with neurodegenerative processes. • In a population-based study cohort, hepatic fat was superior to BMI and visceral and pancreatic fat as a risk biomarker for decreased brain volume of cingulate gyrus and hippocampus. • Increased hepatic fat could serve as a marker of high-risk fat distribution.