Wizard of Oz'ing Technology Demos Needs To Stop

Technology demos and impressive journal publications are flying by at a dizzying rate these days. However, as an industry insider, I recommend maintaining a healthy dose of skepticism about the claimed results datasets the video demonstrations you see. Often what is being presented is the best possible performance. In some cases, what is being shown is actually a performance they hope to soon have which in fact does not yet exist. When this is done, we often call it a Wizard of Oz'ed demonstration. Presenting an aspirational version of what you soon hope to have is oddly close to down right lying.

Back in grad school, I was working on a self-built autonomous vehicle that was to be used for by Ph.D. dissertation work. I got the system up and running and I got my code, which turned camera images into a 3D scene to navigate in, working. I met with a fellow graduate student for dinner that night and I excitedly shared my progress. I was a bit shocked when he asked me, "did you get a video of it?" I indicated I hadn't. He said, "Then it didn't happen." After probing further, I found that his autonomous robotics work which he had recently presented at a robotics conference and was a large part of his doctoral work, had an issue. For every one time his robot autonomously achieved a task, there were another 50 to a hundred times that it would fail. Because of this, he had taken to video taping every single experiment on the off chance the robot would appear to work well, because it could literally be another month or two until another successful trail occurred.

After graduate school my highest paid opportunity came from a major robotics research institute. My first year there, we began preparing for a VIP demo which we spent most of year preparing for. The entire company got less and less sleep as the demonstration approached. Demo day came and went and we each showed our tech demos to the VIPs. They were sufficiently impressed and our funding cycle continued, but each year there was another one or two demos to put together. One thing was for sure, the demos always portrayed an aspirational version of the technology we were developing and the largely non-technical VIPs went away thinking the technology was far more advanced than it actually was.

I will say this R&D cycle removed almost 100% of our ability to do real research. But the phenomenon of misrepresenting the state of your technology to impress investors is a strong driving force for graduate students and companies alike.

When companies demonstrate the technology they have been developing I always ask myself it looks like it has been Wizard of Oz'ed, and you should too. And this applies equally well to results in scientific journal and conference publications as well. The representation you see, is very likely presenting a level of functionality they aspire to more than the actual level of the technology. Even if the data in the tables presented show impressive and improved over state-of-the-art prior work, it is very likely that they have over-fit their algorithms to the datasets they are testing on and that the technology or algorithms performs dramatically less impressively in the wild.

For entertainment, let's keep a list of technology demos and scientific papers that have been called out for doing the old Wizard of Oz trick.

• Tesla's We Robot Event
• Theranos
• ...