Increased cardiorespiratory fitness is related to decreased risk of major chronic illnesses, including cardiovascular disease, type 2 diabetes, and cancer, but its association with colorectal cancer specifically has received very little attention. We examined the relation of cardiorespiratory fitness to colorectal cancer in 59,191 UK Biobank participants aged 39–70 years without prevalent cancer at baseline, followed from 2009 to 2014. Submaximal bicycle ergometry was conducted at study entry, and cardiorespiratory fitness was defined as physical work capacity at 75% of the maximum heart rate, standardised to body mass (PWC75%). Multivariable Cox proportional hazards regression was performed to obtain hazard ratios (HR) and corresponding 95% confidence intervals (CI). During a mean follow-up of 4.6 years, 232 participants developed colorectal cancer (151 colon cancers; 79 rectal cancers). When comparing the 75th to the 25th percentiles of PWC75%, the multivariable-adjusted HR of colorectal cancer was 0.78 (95% CI 0.62–0.97). That relation was largely driven by an inverse association with colon cancer (HR 0.74, 95% CI 0.56–0.97) and less so with rectal cancer (HR 0.88, 95% CI 0.62–1.26; p value for difference by colorectal cancer endpoint = 0.056). The inverse relation of cardiorespiratory fitness with colorectal cancer was more evident in men (HR 0.72, 95% CI 0.55–0.94) than women (HR 0.99, 95% CI 0.71–1.38), although the gender difference was not statistically significant (p value for interaction = 0.192). Increased cardiorespiratory fitness is associated with decreased risk of colorectal cancer. Potential heterogeneity by colorectal cancer anatomic subsite and gender requires further study.