Style Wall : CES Shanghai 2019

Created for Suning to showcase at CES Shanghai 2019, built upon Magic Runway.

In this iteration, we focused solely on the AR clothing, dropping the runway aspect, and building the beginnings of a platform for third party clothing companies to upload and manage their showcase sets. We also created a side kiosk that would allow users to scan a QR code to retrieve photos that they took of themselves while using Style Wall. The QR code serves as a point of entry for the user into Suning’s vast eCommerce websites.

We improved its realism and tracking of the user. We simplified the user interface to allow people to experience AR clothing with less frustration. We implemented lighting tuning tools to control the direction of lighting, lighting exposure and hue so that no matter where Style Wall was set up, it would only take a non-engineer a few minutes to tune the lighting to match the venue.

I improved tracking by further tweaking the Kinect API’s reported joint positions and rotations, which I learned from Magic Runway was unreliable oftentimes. Some joint positions such as the shoulders can be inferred better when they are occluded, rather than freezing in place as the API reports.

I worked closely with our technical artist to redo the base skeleton they used for these new 3D models. The base skeleton better resembles the Kinect skeleton’s expected joint positioning. This fixed a lot of inconsistent stretching and squishing of AR clothing that was sometimes very apparent in Magic Runway (for example, when there is an outfit that has stripes on them). The old 3D models had skeletons that had spines that were curvier and more natural, but Kinect expects very flat torso due to the nature of how it infers joint positions in the torso through a few select joints (shoulders, hips, and center spine).

I also spent a good amount of time fixing foot positioning. Kinect has trouble detecting where the foot meets the floor because from the perspective of the depth sensor, the foot and floor are very close together (similar distance from the camera). This created incorrect leg lengths and jittery positions of the feet reported by the Kinect API.

To correct this, I had to create a tool to allow hard coding of the height of the Kinect, which would need to be measured manually in the real world. Using that height, and the Kinect’s internal orientation, it can infer the floor plane accurately. I can then drop the foot to the floor whenever I detect that the leg is oriented perpendicular to the floor plane, then shift the position of the knee along the leg to match normal human proportions.