We calculate the electronic spectrum of bilayer graphene in perpendicular magnetic fields nonperturbatively. To accommodate arbitrary displacements between the two layers, we apply a periodic gauge based on singular flux vortices of phase 2 pi. The resulting Hofstadter-like butterfly plots show a reduced symmetry, depending on the relative position of the two layers against each other. The split of the zero-energy relativistic Landau level differs by one order of magnitude between Bernal and non-Bernal stacking.