Technology demos and impressive journal publications are flying by at a dizzying rate these days. However, as an industry insider, I recommend maintaining a healthy dose of skepticism about the claimed results datasets the video demonstrations you see. Often what is being presented is the best possible performance. In some cases, what is being shown is actually a performance they hope to soon have which in fact does not yet exist. When this is done, we often call it a Wizard of Oz'ed demonstration. Presenting an aspirational version of what you soon hope to have is oddly close to down right lying.
Back in graduate school, I was working on a self-built autonomous vehicle that was to be used for my Ph.D. dissertation work. I got the system up and running and I got my code, which turned camera images into a 3D scene to navigate in, working. I met with a fellow graduate student for dinner that night and I excitedly shared my progress. He asked me, "did you get a video of it?" I indicated I hadn't and he said, "Then it didn't happen." Obviously I was a bit shocked and had to probe further to understand. He had recently presented his autonomous robotics work at a conference that was a large part of his doctoral work, but he had an issue. For every one time his robot autonomously achieved a task, there were another 50 to a hundred times that it would fail. Because of this, he had taken to making video recordings of every experiment on the off chance the robot would appear to work well, because it could literally be another month or two until another successful trail occurred.
After graduate school my highest paid opportunity came from a major robotics research institute. My first year there, we began preparing for a VIP demo which we spent most of the year working on. The entire company got less and less sleep as the demonstration approached. Demo day came and we each showed our tech demos to the VIPs. They were sufficiently impressed and our funding cycle continued, but each year there was another one or two demos to put together. One thing was for sure, the demos always portrayed an aspirational version of the technology we were developing and the largely non-technical VIPs went away thinking the technology was far more advanced than it actually was.
When companies demonstrate the technology they have been developing I always ask myself if it looks like it has been Wizard of Oz'ed, and you should too. And this applies equally to results in scientific journal and conference publications. The representation you see, is very likely presenting a level of functionality they aspire to more than the actual level of the technology. Even if the data in the tables presented show impressive and improved over state-of-the-art prior work, it is very likely that they have over-fit their algorithms to the datasets they are testing on and that the technology or algorithms perform dramatically less impressively in the wild.
In some ways this is unavoidable because of how much pressure is put on scientists and engineers to show impressive results in order to continue receiving funding. And there is rarely any consequence for the deception because investors rarely do any due diligence or validation of their own. But occasionally a journal paper gets called out for faking their data which cannot be reproduced or a robotics company gets called out for claiming an autonomous demonstration when it was actually teleoperated.
Where do we go from here? Well the public should be much more sceptical of demonstrations and claims and investors should stop wanting to see annual demos because it causes the majority of their investment funds to be spent preparing for the demo. Invest in smart people who are working on something people need, and then leave them to it. If you open the oven early, you could ruin the Soufflé.
© 2024
expert curated independent technology news