Bluetooth RPM Letterboard v2.0

Second iteration of a Rapid Prompting Method (RPM) letterboard with capacitive touch sensors & Bluetooth connectivity

Bluetooth RPM Letterboard

Things used in this project:


Hardware components

  • Adafruit HUZZAH32 – ESP32 Feather Board x 1

  • Adafruit Capacitive Touch Sensor Breakout - MPR121 x 3

  • Adafruit Small Enclosed Piezo w/Wires x 1

  • Adafruit Lithium Ion Polymer Battery - 3.7v 350mAh × 1

  • Adafruit On-Off Power Button / Pushbutton Toggle Switch x 1

  • Adafruit STEMMA QT / Qwiic JST SH 4-pin Cable - 100mm Long x 3

  • Flexible 28AWG Tinned Copper Silicone Stranded Wire x 1

  • Copper Foil Tape with Conductive Acrylic Adhesive x 1

  • M2.5 Nylon Hex Spacer Standoffs Screws Nuts Assortment Kit x 1

Software apps and online services

  • Arduino IDE

  • Espressif Arduino ESP32

  • T-vK's ESP32 BLE Keyboard

Hand tools and fabrication machines

  • Soldering iron (generic)

  • Dupont Crimping Tool Kit

  • Precision Wire Stripper, 30-20 AWG

  • Hot Glue Gorilla Dual Temp

  • LED Lighted Magnifying Third Hand - Five Arms

  • Digital Multimeter

Story


Background


This is the first "production" version of my prototype (www.hackster.io/pedro-martin/bluetooth-rpm-letterboard-7f2ff4).


In a nutshell (full story in the prototype link): The Rapid Prompting Method (RPM) is a technique developed as a vehicle for non-verbal autistic children (and adults) to communicate, that, in effect, unlocks them from a silent isolation and can prove that they are intellectually intact. It can be life-changing for the autistic child and for the people who love them.


RPM uses a letterboard for the student to spell and communicate during the interaction with the facilitator. There are a lot of sensory variables involved. Practice, patience and consistency are only a small portion of what's required.


This letterboard that acts as a Bluetooth keyboard, sending its output to an iOS device that provides typing feedback through its text-to-speech functions.



Design Considerations


The Board


A first requirement is to recognize the selected letter by a very soft touch, so capacitive touch sensing was chosen. This also allows for custom printed faceplates. Also, it has to be very light for comfortable use in one hour sessions (this version weighs in at 200 grams, similar to a small iPhone. Compare to 600 to 900 grams of a Bluetooth keyboard). Finally, it has to survive constant manipulation by the facilitator.


Front View

The Electronics


I chose an ESP32 as MCU because of its speed, BLE and Wi-Fi capabilities (plus a large community for support and libraries). To handle the capacitive touch functions of the letterboard, I chose the MPR121 breakout boards from Adafruit: they are relatively well documented, well developed, and, high quality (as is their HUZZAH ESP32 Feather board). Communication between the three MPR121's (for a total of 36 sensors) and the ESP32 is via I2C. I added a piezo buzzer to provide status feedback in conjunction with the HUZZAH32's built-in red LED. Finally, a 350mAh LiPO battery feeds the ESP32 via a simple On/Off mechanical switch.


Assembly


The base of the board is made from 5mm foam board (rigid enough and lightweight). The front side has 1 sq. inch adhesive copper tape as the touch sensor for each letter (seen here in mid-assembly, with the wires coming in from the back side)


Copper Sensors

The back side has 28 gauge stranded wire leading each copper sensor to the corresponding MPR121 breakout board in which I soldered right angle headers to receive the female Dupont connector of each wire. The position of the MPR121´s allow for the shortest path to each sensor: this minimizes capacitive interference and maintains capacitive sensitivity (see Code Considerations and application note AN3863 in RESOURCES for more information on this topic). The I2C bus and VCC/GND connections are made with STEMMA QT cables.


The HUZZAH32 is soldered to a perfboard for better installation on the foam board. It also has right-angle headers to receive female Dupont terminals. Note that I added on the back side, a 1/2" MDF reinforcement at the top and left sides for better survival of daily manipulation.


Back View

In both cases (front & back), I designed the layout in PowerPoint and printed these layouts on adhesive vinyl which were then added to both sides of the foam board.


PowerPoint Layouts

In order to isolate the I2C & VCC wires (STEMMA QT cables) from the 28 gauge stranded wires connecting the copper sensors to the MPR121's (to avoid noise on both), I added a transparent plastic layer, with the STEMMA QT wires on top and the sensor wires below the plastic layer and glued to the foam board.

Side View

Finally, I added a back cover from the same foam board with a transparent plastic window to see the (green) status LEDs of the MPR121's. I also added a little foam where the hand will hold the board for enhanced comfortability.


Covered Back Side


Code Considerations


To increase the capacitive sensitivity, I changed the CONFIG1 & CONFIG2 registers in Adafruit_MPR121.cpp. This alters the number of samples, the duration of each sample and other associated filtering parameters. The downside would be a reduced available typing speed, that, for this use case, is not a problem. (See the links for application notes AN3889 and AN3890 in "Additional Resources")

//default values = 16uA charge current 6 samples FFI
//writeRegister(MPR121_CONFIG1, 0x10); 

//new values = 32uA charge current 6 samples FFI
writeRegister(MPR121_CONFIG1, 0x20); 

//default values = 0.5uS CDT, 1ms sample interval, 4 samples SFI
//writeRegister(MPR121_CONFIG2, 0x20); 

//New values = 1uS CDT, 1ms period, 10 samples SFI
writeRegister(MPR121_CONFIG2, 0x52);

Periodic Sensor Reset


Any capacitive touch sensor can be affected by changing electromagnetic circumstances in the environment (i.e. humidity, relevant bodies nearby, etc.). Because of the heightened capacitive sensitivity programmed (previous point), these environmental circumstances can negatively impact the response of touch events. To mitigate this, I take advantage of the MPR121 reset function, which recalibrates the sensors for the then existing environmental conditions. So the first reset occurs at minute 3 and every 7 minutes thereafter.


void ResetSensors()  {
  TS1.begin(0x5A); TS2.begin(0x5B); TS3.begin(0x5C);
  TS1.setThresholds(0x19,0x05); 
  TS2.setThresholds(0x19,0x05); 
  TS3.setThresholds(0x19,0x05);
}

void ChkResetCapSensors() {
  int UpTime = int(millis()/1000/60); //--uptime in minutes
  int MinFactor = UpTime % 7; //--divisor is number of minutes between TSs resets
  if (MinFactor != 3) {CapsResetFlag = false;} //--first reset at compared minute
  else 
    if (!CapsResetFlag) {
      ResetSensors();  // Buzz(1);
      CapsResetFlag = true;}
}

System Status Codes


The combination of Piezo Buzzer sounds and HUZZAH32's on-board RED LED provide the following status information


Status Table
void Buzz(int BuzzType) {
  switch (BuzzType) {
    case 1: ledcWriteTone(0,1000); delay(100); ledcWrite(0,0);
            break; //--TS1 is up or scheduled TS reset run
    case 2: ledcWriteTone(0,1500); delay(100); ledcWrite(0,0); 
            break; //--TS2 is up
    case 3: ledcWriteTone(0,2000); delay(100); ledcWrite(0,0); 
            break; //--TS3 is up 
    case 4: ledcWriteTone(0,1100); delay(10); ledcWrite(0,0); delay(1000); 
            digitalWrite(A12, HIGH); ledcWriteTone(0,400); delay(10); ledcWrite(0,0);
            delay(1000); break; //--waiting on BLE pair
    case 5: ledcWriteTone(0,900); delay(300); ledcWriteTone(0,400);
            delay(700); ledcWrite(0,0); break; //--BLE pairing achieved
    case 6: digitalWrite(A12,HIGH); ledcWriteTone(0,100); delay(2000);
            ledcWrite(0,0); break; //--fatal error, must reboot manually
    case 7: ledcWriteTone(0,100); delay(300); ledcWriteTone(0,50);
            delay(700); ledcWrite(0,0); break; //--error, autorestarting TS's
    case 8: digitalWrite(A12,HIGH); ledcWriteTone(0,1500); delay(40);
            ledcWrite(0,0); digitalWrite(A12,LOW); break; //--low battery    
  }
}

Battery Level

After comparing with multimeter readings, the correct battery voltage is achieved applying a correction to the internal pin (A13) reading with this formula:

void ChkBattery() {
  float battRead = analogRead(A13); //--or analogRead(35)
  battRead = ((battRead*2) / 4095 * 3.3 * 1.096);
  if (battRead < 3.3) {  //3.3v as base for beep
    ++counter;
    if (counter > 80) {
      Buzz(8);
      counter = 0;    }}
}

Available Memory on the MCU

To reduce memory requirements, I'm using NimBLE. T-vK's BLE Keyboard reports a bug with the #define statement in ArduinoIDE (but not so in PlatformIO) when using NimBLE. The workaround: Edit BleKeyboard.h and put #define USE_NIMBLE before the first line. Edit BleKeyboard.cpp and move line 17 before the first line.


Another option is to change the HUZZAH32's default memory partition, for example, choosing No OTA or Minimal SPIFFS


Partition Scheme

ESP32 Arduino Core

Releases of Espressif's Arduino Core for ESP32 prior to 2.0 have an issue with I2C. The Arduino IDE will operate with the latest stable version, but take caution that PlatformIO (as of this update) is still using 1.0.6.


Additional Resources


These three application notes from Freescale Semiconductor (manufacturer of the MPR121 chip) are the most relevant for this use case.

AN3889, MPR121 Capacitance Sensing Settings - Application Notes (nxp.com)

AN3890, MPR121 Capacitance Sensing - Filtering and Timing (nxp.com)

AN3863, Designing Touch Sensing Electrodes - Application Notes (nxp.com)

Adafruit's Tutorials provide a good entry point to begin understanding their products

Overview | Adafruit MPR121 12-Key Capacitive Touch Sensor Breakout Tutorial

Overview | Adafruit HUZZAH32 - ESP32 Feather Tutorial

There is no ESPRESSIF or ADAFRUIT implementation of a BLE Keyboard on ESP32 for Arduino IDE, but T-vK's development is extremely useful and stable

T-vK/ESP32-BLE-Keyboard (github.com)

Espressif Arduino Core for ESP32

Getting Started — Arduino-ESP32 2.0.0 documentation (espressif.com)


What's Next


Some of the items I'm exploring to add.

  • Haptic Feedback. To add another sense to the existing two provided by the iOS device (display = eyesight, text-to-speech = hearing), I'm working on a wearable with an Adafruit ItsyBitsy nRF52840 and a Vibrating Mini Motor Disc to produce a Morse-code signature as an additional distinct sensory feedback for each letter (just as each letter has a distinct image and sound). Currently I'm struggling to understand the Arduino Bluefruit nRF52 API for a Dual Role (Central/Peripheral) function with a HID Keyboard service (any pointers are greatly appreciated).

  • On-Board Text-to-Speech. I'm researching how to bypass the iOS device completely by adding a simple 40x2 LCD display and some text-to-speech software, most probably running on a Raspberry Pi Pico or Zero (MCU's may not have the necessary Oomph). Perhaps using Amazon Polly, Festival or Speechelo.(Again, any pointers are greatly appreciated).

  • Visual Approach. Because of the added capacitive sensitivity, as a finger approaches one of the copper sensors (i.e. a letter), it's base measurement begins to drop before it is actually touched. This could be used to gradually light up a small Neopixel to provide added guidance. The issue is potential interference from the Neopixel (as a LED) on the capacitive sensor.

Schematics


Circuit Diagram

Circuit Diagram

Code


RPM Letterboard Code

//---Definitions for Capacitive Touch Sensors
#include <Wire.h>#include <Adafruit_MPR121.h>#ifndef _BV#define _BV(bit) (1 << (bit))#endifAdafruit_MPR121 TS1 = Adafruit_MPR121();Adafruit_MPR121 TS2 = Adafruit_MPR121();Adafruit_MPR121 TS3 = Adafruit_MPR121();uint16_t lastT1, currT1 = 0;uint16_t lastT2, currT2 = 0;uint16_t lastT3, currT3 = 0;int capT, senT = 0;  //--capT=> Touch Sensor Modules (0,1,2), senT=> Individual sensors(0 to 11)bool CapsResetFlag = false;//---Definitions for BLE Keyboard
#define USE_NIMBLE#include <BleKeyboard.h>BleKeyboard bleKeyboard("RPM Letterboard V2.0");bool keybON, action = false;char L[12][3];       //--adding one for null char in char array
//---Other Definitions
int counter = 0;void loadKeyboardMap() {   L[2][0] = 'a'; L[3][0] = 'b';  L[2][2] = 'c'; L[3][2] = 'd';  L[2][1] = 'e'; L[3][1] = '1';   L[1][0] = 'f'; L[4][0] = 'g';  L[1][2] = 'h'; L[4][2] = 'i';  L[1][1] = 'j'; L[4][1] = '2';   L[0][0] = 'k'; L[5][0] = 'l';  L[0][2] = 'm'; L[5][2] = 'n';  L[0][1] = ';'; L[5][1] = '3';  L[11][0] = 'o'; L[6][0] = 'p'; L[11][2] = 'q'; L[6][2] = 'r'; L[11][1] = 's'; L[6][1] = '4';  L[10][0] = 't'; L[7][0] = 'u'; L[10][2] = 'v'; L[7][2] = 'w'; L[10][1] = 'x'; L[7][1] = '5';   L[9][0] = 'y'; L[8][0] = 'z';         L[8][2] = 32;           L[9][1] = 8;   L[8][1] = 10;}void Buzz(int BuzzType) {  switch (BuzzType) {    case 1: ledcWriteTone(0,1000); delay(100); ledcWrite(0,0);            break; //--TS1 is up or scheduled TS reset run
    case 2: ledcWriteTone(0,1500); delay(100); ledcWrite(0,0);            break; //--TS2 is up
    case 3: ledcWriteTone(0,2000); delay(100); ledcWrite(0,0);            break; //--TS3 is up 
    case 4: ledcWriteTone(0,1100); delay(10); ledcWrite(0,0); delay(1000); digitalWrite(A12, HIGH);            ledcWriteTone(0,400); delay(10); ledcWrite(0,0); delay(1000); break; //--waiting on BLE pair
    case 5: ledcWriteTone(0,900); delay(300); ledcWriteTone(0,400);            delay(700); ledcWrite(0,0); break; //--BLE pairing achieved
    case 6: digitalWrite(A12,HIGH); ledcWriteTone(0,100); delay(2000);            ledcWrite(0,0); break; //--fatal error, must reboot manually
    case 7: ledcWriteTone(0,100); delay(300); ledcWriteTone(0,50);            delay(700); ledcWrite(0,0); break; //--error, autorestarting TSs
    case 8: digitalWrite(A12,HIGH); ledcWriteTone(0,1500); delay(40);            ledcWrite(0,0); digitalWrite(A12,LOW); break; //--low battery    
  }}void ResetSensors()  {  TS1.begin(0x5A); TS2.begin(0x5B); TS3.begin(0x5C);  TS1.setThresholds(0x19,0x05); TS2.setThresholds(0x19,0x05); TS3.setThresholds(0x19,0x05);}void ChkResetCapSensors() {  int UpTime = int(millis()/1000/60); //--uptime in minutes
  int MinFactor = UpTime % 7;         //--divisor is number of minutes between TSs resets
  if (MinFactor != 3) {CapsResetFlag = false;}   //--first reset at compared minute
  else    if (!CapsResetFlag) {      ResetSensors();      // Buzz(1);      CapsResetFlag = true;}}void ChkBattery() {  float battRead = analogRead(A13); //--or analogRead(35)  battRead = ((battRead*2) / 4095 * 3.3 * 1.096);  if (battRead < 3.3) {  //3.3v as base for beep
    ++counter;    if (counter > 80) {      Buzz(8);      counter = 0;    }}}void setup() {  //Serial.begin(115200);  pinMode(4, OUTPUT);       //--ledc for Buzzer : Arduino TONE not implemented for ESP32
  ledcSetup(0, 100000, 12); //--Parms=(channel, freq, resolution)  ledcAttachPin(4, 0);  pinMode(A12, OUTPUT);     //--Built in RED LED
  bleKeyboard.begin();  Wire.begin (23, 22);  if (!TS1.begin(0x5A)) {Buzz(6); while (1);} else {Buzz(1);}  if (!TS2.begin(0x5B)) {Buzz(6); while (1);} else {Buzz(2);}  if (!TS3.begin(0x5C)) {Buzz(6); while (1);} else {Buzz(3);}  TS1.setThresholds(0x19,0x05);  TS2.setThresholds(0x19,0x05);  TS3.setThresholds(0x19,0x05);  loadKeyboardMap();  delay(500);}void loop() {  while (!bleKeyboard.isConnected()) {    keybON = false;    digitalWrite(A12, LOW);    Buzz(4); }  if (bleKeyboard.isConnected() && !keybON) {    keybON = true;    Buzz(5);    digitalWrite(A12, LOW); }  ChkResetCapSensors();  ChkBattery();  currT1 = TS1.touched(); currT2 = TS2.touched(); currT3 = TS3.touched();   //--Get the currently touched sensor pads
  for (int i = 0; i < 12; i++) {    if ((TS1.filteredData(i)>10000) or (TS2.filteredData(i)>10000) or (TS3.filteredData(i)>10000)) {      Buzz(7);      ResetSensors(); } //--noise on I2C channel: must reset TSs
    if ((currT1 & _BV(i)) && !(lastT1 & _BV(i))) {capT = 0; senT = i; action = true; }    if ((currT2 & _BV(i)) && !(lastT2 & _BV(i))) {capT = 1; senT = i; action = true; }    if ((currT3 & _BV(i)) && !(lastT3 & _BV(i))) {capT = 2; senT = i; action = true; }}  lastT1 = currT1; lastT2 = currT2; lastT3 = currT3;    //--reset TSs states     
  if (action) {    action = false;    switch (L[senT][capT]) {      case '1':bleKeyboard.print("si "); break;      case '2':bleKeyboard.print("no "); break;      case '3':bleKeyboard.print("verdad "); break;      case '4':bleKeyboard.print("falso "); break;      case '5':bleKeyboard.print("."); break;      default :bleKeyboard.print(L[senT][capT]); break; }}}

-----


This article was originally published on Hackster.io.


Open Minds Silicon Valley provides platforms to elevate the voices of diverse students, professionals, and families. We encourage writing submissions to be emailed to eric@openmindschool.org. We look forward to being in touch about possible feature options.

84 views0 comments