Aging and central vision loss are associated with cortical atrophies, but little is known about the relationship between cortical thinning and the underlying cellular structure. We compared the macro- and micro-structure of the cortical gray and superficial white matter of 38 patients with juvenile (JMD) or age-related (AMD) macular degeneration and 38 healthy humans (19-84 years) by multimodal MRI including diffusion-tensor imaging (DTI). A factor analysis showed that cortical thickness, tissue-dependent measures, and DTI-based measures were sensitive to distinct components of brain structure. Age-related cortical thinning and increased diffusion were observed across most of the cortex, but increased T1-weighted intensities (frontal), reduced T2-weighted intensities (occipital), and reduced anisotropy (medial) were limited to confined cortical regions. Vision loss was associated with cortical thinning and enhanced diffusion in the gray matter (less in the white matter) of the occipital central visual field representation. Moreover, AMD (but not JMD) patients showed enhanced diffusion in lateral occipito-temporal cortex and cortical thinning in the posterior cingulum. These findings demonstrate that changes in brain structure are best quantified by multimodal imaging. They further suggest that age-related brain atrophies (cortical thinning) reflect diverse micro-structural etiologies. Moreover, juvenile and age-related macular degeneration are associated with distinct patterns of micro-structural alterations.