An anomalous signal has been found in Fermi Gamma-Ray Large Area Telescope data covering the center of the Galaxy. Given its morphological and spectral characteristics, this `Galactic Center Excess' is ascribable to self-annihilation of dark matter particles. We report on an analysis that exploits hydrodynamical modeling to register the position of interstellar gas associated with diffuse Galactic gamma-ray emission. Our improved analysis reveals that the excess gamma-rays are spatially correlated with both the X-shaped stellar over-density in the Galactic bulge and the nuclear stellar bulge. Given these correlations, we argue that the Galactic Center gamma-ray excess is not a dark matter phenomenon but rather associated with the stellar population of the bulge and the nuclear bulge.