Soft and responsive colloids based on polymer hydrogels have moved into the focus of the colloid community. This review gives a brief overview of the recent literature on the structure and phase behavior of neutral and ionic poly(N-isopropylacrylamide) microgel dispersions from dilute to over-packed conditions, focusing in particular on the ability of these particles to adapt their size and shape in response to external stimuli. The review is hierarchical; it first covers the aspects of an individual microgel particle viz., the internal structure of inhomogeneous and homogeneously cross-linked particles, followed by studies of ensembles of particles covering in particular structural ordering, phase behavior, and liquid–solid and solid–solid transitions. Insights on the ability of the microgel particles to deform, compress, and interpenetrate beyond the close-packed volume fractions are discussed. Building complex architectures using microgel particles for fundamental studies as well as future applications is reviewed towards the end of the article.