User statistics analysis and split-testing are complementary to testing in the following way: we test to figure out if we implemented what we wanted to implement, and then we run an online user experiment to see if the users like it.
Unfortunately, the machinery needed to make experiment-based decisions is usually very fragile and frequently gets broken causing the whole system to produce incorrect results. In this talk, we’ll review several common problems that exist in split-testing systems. We’ll also provide ways to check if these problems are present in your system.
In 1997-2004, Roman worked in Sun's JDK, in 2004-2008 he moved to Intel where he also worked on Java-related projects. After that he spent 6 years at Yandex, where he gained most of the experience relevant to this talk. Since 2015, Roman changed his professional domain and now trains speakers on technical conferences and generally coaches people willing to make better presentations.