Alzheimer's disease has a long preclinical phase during which amyloid pathology and neurodegeneration accumulate in the brain without producing overt cognitive deficits. It is currently unclear whether these early disease stages are associated with a progressive disruption in the communication between brain regions that subsequently leads to cognitive decline and dementia. In this study we assessed the organization of structural networks in cognitively normal (CN) individuals harboring amyloid pathology (A+N-), neurodegeneration (A-N+), or both (A+N+) from the prospective and longitudinal Swedish BioFINDER study. We combined graph theory with diffusion tensor imaging to investigate integration, segregation, and centrality measures in the brain connectome in the previous groups. At baseline, our findings revealed a disrupted network topology characterized by longer paths, lower efficiency, increased clustering and modularity in CN A-N+ and CN A+N+, but not in CN A+N-. After 2 years, CN A+N+ showed significant abnormalities in all global network measures, whereas CN A-N+ only showed abnormalities in the global efficiency. Network connectivity and organization were associated with memory in CN A+N+ individuals. Altogether, our findings suggest that amyloid pathology is not sufficient to disrupt structural network topology, whereas neurodegeneration is.