Lets talk about robots... by Taylor Stapleton

View fullsize

Ok. So, I can't tell you how many times I have walked into a Walmart or a McDonalds and I just know that the caliber of employee that is going to help me is probably going to be terrible. In fact I often wish that a lot of these service jobs would just be replaced by robots. Sleek, friendly, reliable robots. Well maybe there are some of you out there that would love to just have a robot for a helping hand around the office or for testing as well. Turns out, this dream can actually be a reality (for an enormous sum of money I'm sure). I got the to see a presentation this week about one such company that makes some very specialized kinds of robots just for software testing.

OptoFidelity is a company that makes robots for the purpose of testing mobile phone applications. Essentially, besides an actual human, an OptoFidelity robot is the best you can get for running end to end manual tests on a device. Actually, in the presentation I saw several ways that this robot totally exceeds a humans capability. They seem to have many different form factors of robots with different capabilities. But among the offerings their robots have are:

• Mimic any kind of gestures including different pressure levels and multiple fingers for pinching and zooming.
• Provide very accurate finger hovering distance from screens.
• Do perfect stylus testing including getting angles of stylus and pressure exactly correct.
• Provide an array of sensors by which to measure the devices output (bluetooth, NFC, Wifi) with terrific accuracy.
• Capture the screen with high accuracy cameras.
• Unlike almost any other form of test, the robot could reboot your device manually

One of the main things I was really impressed with is that they had found things to test on a phone that were way outside the scope of what I had imagined. They have a high speed camera watching the phone screen that is accurate enough to perfectly measure the frame rate of the screen at all times. So for instance, the robot finger can touch the screen and the high speed camera will measure exactly how fast and smooth the triggered animation is. So as part of your continuous integration build, you could have a step where you ship the build off to the robot controlled phone. Once on the phone, the robot would do a quick check to see if your application is still as visually fast as it used to be and if you are too slow, your code submission fails. WOW! I WANT THAT!

Apparently as one of their features, they also have the ability to do screen diff-ing and optical character recognition through the camera as well. Although, screen diff-ing through a camera sounds like an insanely hard computer science problem to overcome. So the questions I developed whilst watching this presentation are these:

• How does one define a test suite for a robot? Can it be abstracted to the point where I can write a script for a robot that just gets called by methods like "SwipeUpFromBottom" or "PressTheGreenButton"?
• Can you provide me an interface for which to import the video from the camera into my own software space and use my own image recognition software on the stream?

The Uber Octopusby Taylor Stapleton

I had the opportunity this week to learn first hand about some of the technology Uber uses behind the scenes in testing. While at GTAC I got to meet and hear from Apple Chow (Engineering Manager) and Bian Jiang (Senior Software Engineer) of the Mobile Infra Team at Uber. Before I move on, I would like to acknowledge that yes, you read that correctly. There is a human female named Apple. Well, ol' Apple and the gang claim to have a unique testing problem to solve in the way that Uber really has two apps instead of one. Uber has the driver application and the rider application and in some cases, there will be a combination of many rider applications interacting with one driver application (think Uber Pool). To solve their "unique" problem they developed a tool called Octopus. One could imagine an octopus with its many legs controlling many phones concurrently.

So the point of Octopus is that it can help handle complex end to end test scenarios where they want to test the interaction between multiple applications. The kind of test this is meant for would be something along the lines of something that would be well suited for a manual test where two QA people interacted with each other on the device or something. The main feature of Octopus is that it has a way for applications running on devices or in emulators to be able to talk to one another. Specifically, test runners running on each instance can talk to each other to have one big collaborative test scenario per se. The communication they have developed is branded in the Octopus tools as "signals" and the writer of the test scenarios is provided with two main API's. readSignal and writeSignal. Reading a signal naturally is blocking, and so that helps to control the flow of the tests. In case its a little hard to picture, depicted below would be an example test scenario.

I didn't get the full story of how it's implemented on the back end, but here are a few details I was able to gather from the Uber engineers:

• It works for both Android and IOS applications.
• When running on physical devices, it is necessary for the physical devices to be connected to a special Octopus hardware hub.
• When running in an emulator it is necessary for the hardware hub to also be emulated through use of writing to various files.

So let's talk about this solution. My first thought is that it seems like a pretty expensive solution to being able to run two applications in parallel for a test. There is a lot of software going into this and on top of that, there is a dedicated piece of hardware that sits in the middle of devices that must have required significant development as well. I get the feeling that there must be some easier ways to implement a test runner that can run scenarios between multiple applications. Even just writing out simple key/value pairs via http to a very simple server somewhere seems preferable. Maybe I don't know enough about the value proposition of the tool. :-p

Despite my skepticism though, there are some pretty cool perks to having the integrated test scenarios running, however the tool is implemented. When you actually see the demo they give of the test runner booting up 5 emulators and then performing UI actions on all the different applications, its pretty cool. In case it isn't immediately clear what this inter-test interaction is good for, allow me to explain. One blatant example I can think of: as an Uber rider, you would know when your car has arrived because a physical car would pull up at your pickup location. In an automated test scenario however, this isn't quite possible. How does the rider test application know when to continue the test scenario and acknowledge that the driver has arrived? One option would be to rely on certain UI elements but there are just so many hard problems to solve when doing any kind of element testing or screen diff-ing.

Having some backend way for tests to send signals to each other is a pretty good idea. Though I'm not so sure about the implementation (seems complicated), the overall effect of the signals is pretty cool. Hats off to Apple, Bian, and the other Uber engineers that built Octopus. And if any of you someday discover this post and want to leave a comment on why you did things the way you did, I would be thrilled to know.