Make tangible learning inexpensive
Physical learning systems often depend on tablets, computers, proprietary hardware, or cloud services. PicoTact explores how much of that experience can instead run on inexpensive embedded hardware.
Low-cost embedded vision for tangible learning experiences.
Overview
PicoTact is a roughly $25 standalone embedded vision platform for making physical learning materials interactive. It combines an ESP32-CAM, camera, display, and lightweight computer vision so activities can respond to what students do in the physical world. Like Osmo, the commercial learning system that uses a tablet's camera to recognize and respond to physical tangible objects the students manipulate, PicoTact aims to provide a similar experience.
Pico means small, while tact refers to tactile or tangible interaction. The project is designed around a small, standalone device for tangible learning. Earlier work, including A General-Purpose Vision and Audio Feedback Pipeline for Camera-Observed Physical Puzzles in K–5 Learning, serves as a basis for this project.
The same platform can support activities such as tangrams, geography, spelling, counting, and tangible coding without requiring a tablet, external computer, or cloud service. The broader goal is to bring an Osmo-like learning experience to the most scalable audience possible.
Embedded Vision
Education
Tangible Interaction
ESP32
Research question
How do we bring an Osmo-like, tangible experience to a scalable, embedded and resource effective platform?
Goals
Make one platform support many activities
Rather than building a separate electronic toy for every lesson, PicoTact uses vision to understand ordinary physical artifacts. The same device can support activities involving shapes, words, maps, numbers, and other manipulatives.
Keep the computation local
The system is designed to detect and respond to physical interactions directly on the device. Lightweight vision keeps the platform usable without internet access or expensive AI hardware.
Interaction model
Student workflow
Place tangible object
Camera (ESP32CAM) detects it in space and reasons
Embedded vision pipeline (homography, masking)
Update screen for real-time interaction based on detection
PicoTact in action
Note: These examples were tested using the vision pipeline described in this paper, running on a Pi-class device. Development of the same pipeline for PicoTact’s ESP32 version is currently in progress.
Bird
Bird tangram game. One of the many tangrams you can build, based on the style of Osmo’s flagship Tangram game.
Pizza
Pizza building game. Draw a circle, then build the toppings of your pizza using shape detection.
Shape
Shape detection game. Place the correct shape that the player wants to eat. This one is the most real-time example, as the system continuously keeps track of the interaction.
Square
Building shapes. Build a square using ice cream rods while the system detects the pieces and checks the completed shape.
Technologies used
ESP32 hardware stack
PicoTact runs on an ESP32 with an overhead OV5640 camera and an SPI display. Local storage keeps the activity data on the device. The prototype is housed above the interaction area and currently costs about $25 using off the shelf parts. All processing happens locally.
ESP32 Website ↗Classical vision pipeline
PicoTact uses classical computer vision rather than a large neural model. Frames are cleaned and thresholded before the system finds contours and connected regions. It then estimates the shape, position and orientation of each object. A calibrated board maps camera coordinates to the activity space, while color, geometry, size, templates or markers can be used to identify pieces.
OV5640 Reference ↗Embedded firmware
The device runs embedded C and C++ software that connects the full interaction loop. A physical action is captured by the camera, processed locally, matched to the current activity state and then reflected on the SPI display. Different activities can reuse the same hardware by changing the detection rules and screen content.
Vision pipeline in paper ↗More of my work
