Neuroanatomical differences attributable to aging and gender have been well documented, and these differences may be associated with differences in behaviors and cognitive performance. However, little is known about the dynamic organization of anatomical connectivity within the cerebral cortex, which may underlie population differences in brain function. In this study, we investigated age and sex effects on the anatomical connectivity patterns of 95 normal subjects ranging in age from 19 to 85 years. Using the connectivity probability derived from diffusion magnetic resonance imaging tractography, we characterized the cerebral cortex as a weighted network of connected regions. This approach captures the underlying organization of anatomical connectivity for each subject at a regional level. Advanced graph theoretical analysis revealed that the resulting cortical networks exhibited “small-world” character (i.e., efficient information transfer both at local and global scale). In particular, the precuneus and posterior cingulate gyrus were consistently observed as centrally connected regions, independent of age and sex. Additional analysis revealed a reduction in overall cortical connectivity with age. There were also changes in the underlying network organization that resulted in decreased local efficiency, and also a shift of regional efficiency from the parietal and occipital to frontal and temporal neocortex in older brains. In addition, women showed greater overall cortical connectivity and the underlying organization of their cortical networks was more efficient, both locally and globally. There were also distributed regional differences in efficiency between sexes. Our results provide new insights into the substrates that underlie behavioral and cognitive differences in aging and sex.