Hello! For my project I chose the Gesture Controlled Robot! It is a robot car that is controlled by gestures created by a hand module. The robot and hand are connected with bluetooth, and programmed on Arduino IDE with C++. An accelerometer on the hand module measures tilts in the x and y direction, prompting the robot car to move it’s motors in certain directions.
| Engineer | School | Area of Interest | Grade |
|---|---|---|---|
| Mari I | Forest Hills High School | Electrical Engineering | Incoming Senior |
Final Milestone
Since my last milestone I have fully completed the base project! The robot and hand module operate wirelessly with no issues. For modifications I created a glove to place the hand module on. This is so that I can wear it, and it looked cooler! Another modification I added was a camera and screen. I planned to have a ESP-32 Camera mounted on the robot to stream where the robot was moving, and a ILI9341 TFT screen to display it from a distance.
The most challenging aspect of this milestone was debugging the screen and camera. I had to create its own wifi network with 2.4Ghz, and configuring between the camera and display was excruciatingly hard. Whether it was finding the right resolution, quality or frame rate I went through hundreds of lines of code trying to find that one error that was distorting my screen. In the end, I was able to find the bugs and have my screen display the camera’s live footage. This took a toll on my mental state was well as my hope but It was able to teach a great lesson of perseverance. The struggles I faced throughout this project made me realize how much I love the satisfaction of overcoming it.
Second Milestone
Since my last milestone, I successfully connected the Bluetooth modules so the hand controller can communicate wirelessly with the robot. I also added an accelerometer to the controller, allowing it to detect my hand movements. After sending the X and Y coordinates to the robot, I programmed the Arduino Uno to respond to those values and move accordingly. To make the controller completely portable, I powered it with a 9-volt battery.
The biggest challenges were wiring the electronics, getting the Bluetooth modules to communicate, and debugging the accelerometer. For my next milestone, I plan to turn the controller into a wearable glove and add a speed boost feature to the robot to improve its performance. Thank you for watching!
First Milestone
For my first milestone, I focused mainly on the hardware. I followed the tutorial to build the robot chassis by attaching the wheels and motors. I also wired the motors, motor driver, and Arduino UNO together. I soon realized that the battery clips had the wrong connector to attach to my motor driver. While I was waiting for the correct parts to arrive, I was able to test each motor individually. Then, to get the Arduino UNO to control the motor driver, I wrote a sketch that allowed the motors to move forward. To power the UNO, I used the battery pack with the barrel jack attachment.
To become more familiar with C++, I programmed my robot to perform a sequence of movements: repeat four times by moving forward and then turning right, perform a spin, move backward, and then move forward again. Once my new battery clip arrived, I completed many trial runs to perfect the sequence.
That is my progress so far! Next, I would like to work on connecting the Bluetooth modules and perfecting the hand module.
Robot Car Schematics
Hand Module Schematics
Hand Module Code
#include <Wire.h>
#define BT_Serial Serial1
const int MPU_ADDR = 0x68;
int16_t AcX, AcY, AcZ;
int X_value;
int Y_value;
int flag = 0;
void setup() {
Serial.begin(38400); // Serial Monitor
BT_Serial.begin(38400); // HC-05
Wire.begin();
// Wake up MPU6050
Wire.beginTransmission(MPU_ADDR);
Wire.write(0x6B);
Wire.write(0);
Wire.endTransmission(true);
delay(500);
Serial.println("MPU6050 Ready");
}
void loop() {
readMPU6050();
Serial.print("X: ");
Serial.print(X_value);
Serial.print(" Y: ");
Serial.println(Y_value);
// Forward (tilt forward farther)
if (X_value < 45 && flag == 0) {
flag = 1;
BT_Serial.write('f');
Serial.println("Forward");
}
// Backward (tilt backward farther)
else if (X_value > 135 && flag == 0) {
flag = 1;
BT_Serial.write('b');
Serial.println("Backward");
}
// Left
else if (Y_value < 45 && flag == 0) {
flag = 1;
BT_Serial.write('l');
Serial.println("Left");
}
// Right
else if (Y_value > 135 && flag == 0) {
flag = 1;
BT_Serial.write('r');
Serial.println("Right");
}
// Larger neutral zone = stop
else if (X_value > 60 && X_value < 120 &&
Y_value > 60 && Y_value < 120 &&
flag == 1) {
flag = 0;
BT_Serial.write('s');
Serial.println("Stop");
}
delay(100);
}
void readMPU6050() {
Wire.beginTransmission(MPU_ADDR);
Wire.write(0x3B);
Wire.endTransmission(false);
Wire.requestFrom(MPU_ADDR, 6, true);
AcX = Wire.read() << 8 | Wire.read();
AcY = Wire.read() << 8 | Wire.read();
AcZ = Wire.read() << 8 | Wire.read();
// Convert raw accelerometer values
X_value = map(AcX, -17000, 17000, 0, 180);
Y_value = map(AcY, -17000, 17000, 0, 180);
X_value = constrain(X_value, 0, 180);
Y_value = constrain(Y_value, 0, 180);
}
Robot Car Code
#include <SoftwareSerial.h>
SoftwareSerial BT_Serial(2, 3); // RX, TX
// Motor Driver Pins
#define ENA 5
#define IN1 6
#define IN2 7
#define ENB 10
#define IN3 8
#define IN4 9
char bt_data = 's';
int Speed = 180;
void setup()
{
Serial.begin(9600);
BT_Serial.begin(38400);
pinMode(ENA, OUTPUT);
pinMode(IN1, OUTPUT);
pinMode(IN2, OUTPUT);
pinMode(ENB, OUTPUT);
pinMode(IN3, OUTPUT);
pinMode(IN4, OUTPUT);
Stop();
delay(500);
Serial.println("Robot Ready");
}
void loop()
{
if (BT_Serial.available() > 0)
{
bt_data = BT_Serial.read();
Serial.print("Received: ");
Serial.println(bt_data);
}
if (bt_data == 'f')
{
forward();
Speed = 180;
}
else if (bt_data == 'b')
{
backward();
Speed = 180;
}
else if (bt_data == 'l')
{
turnLeft();
Speed = 120;
}
else if (bt_data == 'r')
{
turnRight();
Speed = 120;
}
else if (bt_data == 's')
{
Stop();
}
analogWrite(ENA, Speed);
analogWrite(ENB, Speed);
delay(50);
}
// Forward
void forward()
{
digitalWrite(IN1, HIGH);
digitalWrite(IN2, LOW);
digitalWrite(IN3, LOW);
digitalWrite(IN4, HIGH);
}
// Backward
void backward()
{
digitalWrite(IN1, LOW);
digitalWrite(IN2, HIGH);
digitalWrite(IN3, HIGH);
digitalWrite(IN4, LOW);
}
// Turn Right (switched)
void turnRight()
{
digitalWrite(IN1, HIGH);
digitalWrite(IN2, LOW);
digitalWrite(IN3, HIGH);
digitalWrite(IN4, LOW);
}
// Turn Left (switched)
void turnLeft()
{
digitalWrite(IN1, LOW);
digitalWrite(IN2, HIGH);
digitalWrite(IN3, LOW);
digitalWrite(IN4, HIGH);
}
//Stop
void Stop()
{
digitalWrite(IN1, LOW);
digitalWrite(IN2, LOW);
digitalWrite(IN3, LOW);
digitalWrite(IN4, LOW);
}
Camera Code
#include "esp_camera.h"
#include <WiFi.h>
#include <ArduinoWebsockets.h>
#define CAMERA_MODEL_AI_THINKER
#include "camera_pins.h"
const char* ssid = "esp32net";
const char* password = "123456789";
const char* websockets_server_host = "192.168.4.1";
const uint16_t websockets_server_port = 80;
using namespace websockets;
WebsocketsClient client;
void setup() {
Serial.begin(115200);
Serial.setDebugOutput(true);
Serial.println();
camera_config_t config;
config.ledc_channel = LEDC_CHANNEL_0;
config.ledc_timer = LEDC_TIMER_0;
config.pin_d0 = Y2_GPIO_NUM;
config.pin_d1 = Y3_GPIO_NUM;
config.pin_d2 = Y4_GPIO_NUM;
config.pin_d3 = Y5_GPIO_NUM;
config.pin_d4 = Y6_GPIO_NUM;
config.pin_d5 = Y7_GPIO_NUM;
config.pin_d6 = Y8_GPIO_NUM;
config.pin_d7 = Y9_GPIO_NUM;
config.pin_xclk = XCLK_GPIO_NUM;
config.pin_pclk = PCLK_GPIO_NUM;
config.pin_vsync = VSYNC_GPIO_NUM;
config.pin_href = HREF_GPIO_NUM;
config.pin_sscb_sda = SIOD_GPIO_NUM;
config.pin_sscb_scl = SIOC_GPIO_NUM;
config.pin_pwdn = PWDN_GPIO_NUM;
config.pin_reset = RESET_GPIO_NUM;
config.xclk_freq_hz = 20000000;
config.pixel_format = PIXFORMAT_JPEG;
//init with high specs to pre-allocate larger buffers
if(psramFound()){
config.frame_size = FRAMESIZE_QVGA; // 320x240
config.jpeg_quality = 20;
config.fb_count = 2;
} else {
config.frame_size = FRAMESIZE_QVGA;
config.jpeg_quality = 20;
config.fb_count = 1;
}
// camera init
esp_err_t err = esp_camera_init(&config);
if (err != ESP_OK) {
Serial.printf("Camera init failed with error 0x%x", err);
return;
}
sensor_t *s = esp_camera_sensor_get();
s->set_vflip(s, 1);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.print("Camera Ready! Use 'http://");
Serial.print(WiFi.localIP());
Serial.println("' to connect");
while(!client.connect("ws://192.168.4.1:80")){
delay(500);
Serial.print(".");
}
Serial.println("Socket Connected!");
}
void loop() {
camera_fb_t *fb = NULL;
esp_err_t res = ESP_OK;
fb = esp_camera_fb_get();
if(!fb){
Serial.println("Camera capture failed");
esp_camera_fb_return(fb);
return;
}
size_t fb_len = 0;
if(fb->format != PIXFORMAT_JPEG){
Serial.println("Non-JPEG data not implemented");
return;
}
Serial.printf("JPEG = %d bytes\n", fb->len);
client.sendBinary((const char*)fb->buf, fb->len);
esp_camera_fb_return(fb);
delay(50);
}
Display Screen Code
#include <SPI.h>
#include <ArduinoWebsockets.h>
#include <WiFi.h>
#include <TJpg_Decoder.h>
#include <TFT_eSPI.h>
const char* ssid = "esp32net";
const char* password = "123456789";
using namespace websockets;
WebsocketsServer server;
WebsocketsClient client;
TFT_eSPI tft = TFT_eSPI(); // Invoke custom library
bool tft_output(int16_t x, int16_t y, uint16_t w, uint16_t h, uint16_t* bitmap)
{
// Stop further decoding as image is running off bottom of screen
if ( y >= tft.height() ) return 0;
// This function will clip the image block rendering automatically at the TFT boundaries
tft.pushImage(x, y, w, h, bitmap);
// This might work instead if you adapt the sketch to use the Adafruit_GFX library
// tft.drawRGBBitmap(x, y, bitmap, w, h);
// Return 1 to decode next block
return 1;
}
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
delay(1000);
tft.begin();
Serial.printf("Display = %d x %d\n", tft.width(), tft.height());
tft.fillScreen(TFT_RED);
delay(1000);
tft.fillScreen(TFT_GREEN);
delay(1000);
tft.fillScreen(TFT_BLUE);
delay(1000);
tft.setRotation(3);
tft.setTextColor(0xFFFF, 0x0000);
tft.fillScreen(TFT_RED);
tft.setSwapBytes(true); // We need to swap the colour bytes (endianess)
// The jpeg image can be scaled by a factor of 1, 2, 4, or 8
TJpgDec.setJpgScale(1);
// The decoder must be given the exact name of the rendering function above
TJpgDec.setCallback(tft_output);
Serial.println();
Serial.println("Setting AP...");
WiFi.softAP(ssid, password);
IPAddress IP = WiFi.softAPIP();
Serial.print("AP IP Address : ");
Serial.println(IP);
server.listen(80);
}
void loop() {
if(server.poll()){
client = server.accept();
}
if(client.available()){
client.poll();
WebsocketsMessage msg = client.readBlocking();
Serial.printf("Received = %d bytes\n", msg.length());
uint32_t t = millis();
// Get the width and height in pixels of the jpeg if you wish
uint16_t w = 0, h = 0;
TJpgDec.getJpgSize(&w, &h, (const uint8_t*)msg.c_str(), msg.length());
Serial.print("Width = "); Serial.print(w); Serial.print(", height = "); Serial.println(h);
// Draw the image, top left at 0,0
TJpgDec.drawJpg(0, 0, (const uint8_t*)msg.c_str(), msg.length());
// How much time did rendering take (ESP8266 80MHz 271ms, 160MHz 157ms, ESP32 SPI 120ms, 8bit parallel 105ms
t = millis() - t;
Serial.print(t); Serial.println(" ms");
}
}
Bill of Materials
| Part | Note | Price | Link |
|---|---|---|---|
| Car Chassis Kit | Wheels and Base | $39.99 | Link |
| Screwdriver Kit | Putting together parts | $5.94 | Link |
| Arduino Uno Clone | Microcontroller for car | $14.98 | Link |
| Electronics Kit | Parts for electornics | $14 | Link |
| Breadboard Kit | Connecting electronic parts & wires | $8.79 | Link |
| Arduino Nano 33 BLE Sense | Microcontroller for hand module | $39.7 | Link |
| Mirco USB Cable | Connecting to computer | $5 | Link |
| Accelerometer | Measure tilt & motion in hand module | $9 | Link |
| HC05 | Bluetooth conections | $9 | Link |
| Breadboard Power Supply | Supply power to breadboards | $8 | Link |
| 9v Batteries | Power the Arduino UNO and Nano | $8.69 | Link |
| Velcro Tape | Strapping and securing | $8 | Link |
| DMM | Measures Voltage | $9.99 | Link |