Dimensionality crossover to a two-dimensional vestigial nematic state from a three-dimensional antiferromagnet in a honeycomb van der Waals magnet
The effects of fluctuations and disorder, which are substantially enhanced in reduced dimensional-ities, can play a crucial role in producing nontrivial phases of matter such as vestigial orders char-acterized by a composite order parameter. However, fluctuation-driven magnetic phases in low dimensions have remained relatively unexplored. Here we demonstrate a phase transition from the zigzag antiferromagnetic order in the three-dimensional bulk to a Z 3 vestigial Potts-nematicity in two-dimensional few-layer samples of van der Waals magnet NiPS 3 . Our spin relaxometry and op-tical spectroscopy measurements reveal that the spin fluctuations are enhanced over the gigahertz to terahertz range as the layer number of NiPS 3 reduces. Monte Carlo simulations corroborate the experimental finding of 3-fold rotational symmetry breaking, but show that translational symmetry is restored in thin layers of NiPS 3 . Furthermore, our results show that strong quantum fluctuations can stabilize an unconventional magnetic phase after destroying a more conventional one.