Modern service based software is developed in short cycles with continuous user feedback. My research interests are in building tools that support this process and empirical evaluations of their use. The tools use machine learning, combinatorial optimization, and domain specific languages to specify software configurations that should be optimized with user data. The tools must be evaluated in realistic environments involving real end user data at industry sites.