The Dreams of IoT is a smart home projection installation augmented with connected sensors’ imaginaries through real-time GANs image generation. Users can also control different sensors separately through multi-platform web pages in order to interconnect items with the network. This project is exhibited in ITP Spring Show 2022.
Relevant Tools
C++, Arduino, node.js, HTML/CSS, Javascript, Fusion 360, Figma
My role
Programmer, Full-stack developer
Teammates
Zeshu Zhu, Meijie Hu
IDEATION
In the book The Dream of The Internet of Things by Alice Benessia and Ângela Guimarães Pereira, technology, particularly in the context of IoT, “experience can be replaced by a series of algorithmic instructions, designed by software and hardware developers”. However, it is rather human-centric (and boring!) to think that machines cannot think creatively with the information it receives. Especially with the advancement of machine learning, our traditional conception that devices lack imagination is further challenged.
Therefore, we recreate a few common smart home sensors such as temperature sensors, light sensors, and motion detectors to collect data in the room, and we translate them into dream imageries with GANs and map them across the room. The whole smart home system is connected through a server and monitored through a central platform. We hope the Dreams of the Internet of Things can demonstrate the power of technology behind smart homes, and we prompt participants to reconsider connected devices in a more informative yet imaginative way when they are immersed in the dreams weaved by the sensors from their daily life.
Challenges
To achieve this we face many challenges:
- GANs are generated very slow, even if we generate low-resolution images a picture still takes tens of seconds, which is difficult to achieve the “real-time” we want.
- The second is the synchronization of multiple programming environments. GANs are mainly in Python, while the Sensor we use is mainly in C++, and the web pages we want to render will be mainly in JavaScript, which means we need to build a way to link three programming languages in very different environments to communicate with each other.
- The next design challenge is how to relate the sensor to its function. Since this is for home use, we have to consider that its design must bring the user a good experience while also allowing the user to understand its function.
- The final challenge is the user experience. This is a product with complex functions, how do we simplify its complex functions so that users can easily use it without understanding any principles.
PROCESS
For the first, second and fourth challenges, we decided to create a web page for the user to manipulate and review the sensor, believing that with good web design, the user could do with just three to four buttons.
To achieve real-time, we decided to preload the data in advance, which means that we preload tens of seconds of data before the user opens it for use and use the preloaded data before generating the next image. When the image is available and can be replenished without an image, we can use the real-time data.
Website Design
First draft
In our first draft, we designed to highlight the generated content and allow the user to jump to the corresponding sensor by clicking on the link. We expected users only have two options to choose between switching sensors pages and connected/disconnected.
We got a lot of feedback from users after using our webpage:
- Many users think that frequent page switching is unnecessary because they prefer to get all the information on one page rather than switching.
- Users think we don’t need to put the generated image in the background to emphasize it, because eventually we will use projections and an overly large background will not only be distracting, but will more often make users notice the content of the background more than the action buttons.
- Many users do not know what to do when they open the page: what is clickable and what information is updated in real time. All of these issues plague users.
iteration
After getting user feedback, we decided to change the overall design to a one-page design instead of a multi-page design, and used different UI for different buttons, so users could understand how to operate faster. We condensed the content of the different sensors into different squares of the seciton, so that instead of jumping from page to page, users can view and control all the content in one page.
We have also improved the compatibility of the web pages, not only on computers, but also on tablets and cell phones, so that users can access and use the pages.
Hardware Design
We used Arduino nano as our micro controller and added screens to some of the sensors for better user access to information. And we built in a rechargeable battery to ensure that the user can use it anywhere and recharge it quickly when it runs out of power.
