Bluetooth RPM Letterboard v2.0

Second iteration of a Rapid Prompting Method (RPM) letterboard with capacitive touch sensors & Bluetooth connectivity

Hardware components

Software apps and online services

Hand tools and fabrication machines

This is the first "production" version of my prototype (www.hackster.io/pedro-martin/bluetooth-rpm-letterboard-7f2ff4).

In a nutshell (full story in the prototype link): The Rapid Prompting Method (RPM) is a technique developed as a vehicle for non-verbal autistic children (and adults) to communicate, that, in effect, unlocks them from a silent isolation and can prove that they are intellectually intact. It can be life-changing for the autistic child and for the people who love them.

RPM uses a letterboard for the student to spell and communicate during the interaction with the facilitator. There are a lot of sensory variables involved. Practice, patience and consistency are only a small portion of what's required.

This letterboard that acts as a Bluetooth keyboard, sending its output to an iOS device that provides typing feedback through its text-to-speech functions.

The Board

A first requirement is to recognize the selected letter by a very soft touch, so capacitive touch sensing was chosen. This also allows for custom printed faceplates. Also, it has to be very light for comfortable use in one hour sessions (this version weighs in at 200 grams, similar to a small iPhone. Compare to 600 to 900 grams of a Bluetooth keyboard). Finally, it has to survive constant manipulation by the facilitator.

The Electronics

I chose an ESP32 as MCU because of its speed, BLE and Wi-Fi capabilities (plus a large community for support and libraries). To handle the capacitive touch functions of the letterboard, I chose the MPR121 breakout boards from Adafruit: they are relatively well documented, well developed, and, high quality (as is their HUZZAH ESP32 Feather board). Communication between the three MPR121's (for a total of 36 sensors) and the ESP32 is via I2C. I added a piezo buzzer to provide status feedback in conjunction with the HUZZAH32's built-in red LED. Finally, a 350mAh LiPO battery feeds the ESP32 via a simple On/Off mechanical switch.

The base of the board is made from 5mm foam board (rigid enough and lightweight). The front side has 1 sq. inch adhesive copper tape as the touch sensor for each letter (seen here in mid-assembly, with the wires coming in from the back side)

The back side has 28 gauge stranded wire leading each copper sensor to the corresponding MPR121 breakout board in which I soldered right angle headers to receive the female Dupont connector of each wire. The position of the MPR121´s allow for the shortest path to each sensor: this minimizes capacitive interference and maintains capacitive sensitivity (see Code Considerations and application note AN3863 in RESOURCES for more information on this topic). The I2C bus and VCC/GND connections are made with STEMMA QT cables.

The HUZZAH32 is soldered to a perfboard for better installation on the foam board. It also has right-angle headers to receive female Dupont terminals. Note that I added on the back side, a 1/2" MDF reinforcement at the top and left sides for better survival of daily manipulation.

In both cases (front & back), I designed the layout in PowerPoint and printed these layouts on adhesive vinyl which were then added to both sides of the foam board.

In order to isolate the I2C & VCC wires (STEMMA QT cables) from the 28 gauge stranded wires connecting the copper sensors to the MPR121's (to avoid noise on both), I added a transparent plastic layer, with the STEMMA QT wires on top and the sensor wires below the plastic layer and glued to the foam board.

Finally, I added a back cover from the same foam board with a transparent plastic window to see the (green) status LEDs of the MPR121's. I also added a little foam where the hand will hold the board for enhanced comfortability.

Code Considerations

To increase the capacitive sensitivity, I changed the CONFIG1 & CONFIG2 registers in Adafruit_MPR121.cpp. This alters the number of samples, the duration of each sample and other associated filtering parameters. The downside would be a reduced available typing speed, that, for this use case, is not a problem. (See the links for application notes AN3889 and AN3890 in "Additional Resources")

//default values = 16uA charge current 6 samples FFI
 //writeRegister(MPR121_CONFIG1, 0x10);
 
 //new values = 32uA charge current 6 samples FFI
 writeRegister(MPR121_CONFIG1, 0x20);
 
 //default values = 0.5uS CDT, 1ms sample interval, 4 samples SFI
 //writeRegister(MPR121_CONFIG2, 0x20);
 
 //New values = 1uS CDT, 1ms period, 10 samples SFI
 writeRegister(MPR121_CONFIG2, 0x52);
 

Periodic Sensor Reset

Any capacitive touch sensor can be affected by changing electromagnetic circumstances in the environment (i.e. humidity, relevant bodies nearby, etc.). Because of the heightened capacitive sensitivity programmed (previous point), these environmental circumstances can negatively impact the response of touch events. To mitigate this, I take advantage of the MPR121 reset function, which recalibrates the sensors for the then existing environmental conditions. So the first reset occurs at minute 3 and every 7 minutes thereafter.

void ResetSensors() {
  TS1.begin(0x5A); TS2.begin(0x5B); TS3.begin(0x5C);
  TS1.setThresholds(0x19,0x05);
  TS2.setThresholds(0x19,0x05);
  TS3.setThresholds(0x19,0x05);
 }
 
 void ChkResetCapSensors() {
  int UpTime = int(millis()/1000/60); //--uptime in minutes
  int MinFactor = UpTime % 7; //--divisor is number of minutes between TSs resets
  if (MinFactor != 3) {CapsResetFlag = false;} //--first reset at compared minute
  else
  if (!CapsResetFlag) {
  ResetSensors(); // Buzz(1);
  CapsResetFlag = true;}
 }
 

System Status Codes

The combination of Piezo Buzzer sounds and HUZZAH32's on-board RED LED provide the following status information

void Buzz(int BuzzType) {
  switch (BuzzType) {
  case 1: ledcWriteTone(0,1000); delay(100); ledcWrite(0,0);
  break; //--TS1 is up or scheduled TS reset run
  case 2: ledcWriteTone(0,1500); delay(100); ledcWrite(0,0);
  break; //--TS2 is up
  case 3: ledcWriteTone(0,2000); delay(100); ledcWrite(0,0);
  break; //--TS3 is up
  case 4: ledcWriteTone(0,1100); delay(10); ledcWrite(0,0); delay(1000);
  digitalWrite(A12, HIGH); ledcWriteTone(0,400); delay(10); ledcWrite(0,0);
  delay(1000); break; //--waiting on BLE pair
  case 5: ledcWriteTone(0,900); delay(300); ledcWriteTone(0,400);
  delay(700); ledcWrite(0,0); break; //--BLE pairing achieved
  case 6: digitalWrite(A12,HIGH); ledcWriteTone(0,100); delay(2000);
  ledcWrite(0,0); break; //--fatal error, must reboot manually
  case 7: ledcWriteTone(0,100); delay(300); ledcWriteTone(0,50);
  delay(700); ledcWrite(0,0); break; //--error, autorestarting TS's
  case 8: digitalWrite(A12,HIGH); ledcWriteTone(0,1500); delay(40);
  ledcWrite(0,0); digitalWrite(A12,LOW); break; //--low battery
  }
 }

Battery Level

After comparing with multimeter readings, the correct battery voltage is achieved applying a correction to the internal pin (A13) reading with this formula:

void ChkBattery() {
  float battRead = analogRead(A13); //--or analogRead(35)
  battRead = ((battRead*2) / 4095 * 3.3 * 1.096);
  if (battRead < 3.3) { //3.3v as base for beep
  ++counter;
  if (counter > 80) {
  Buzz(8);
  counter = 0; }}
 }

Available Memory on the MCU

To reduce memory requirements, I'm using NimBLE. T-vK's BLE Keyboard reports a bug with the #define statement in ArduinoIDE (but not so in PlatformIO) when using NimBLE. The workaround: Edit BleKeyboard.h and put #define USE_NIMBLE before the first line. Edit BleKeyboard.cpp and move line 17 before the first line.

Another option is to change the HUZZAH32's default memory partition, for example, choosing No OTA or Minimal SPIFFS

ESP32 Arduino Core

Releases of Espressif's Arduino Core for ESP32 prior to 2.0 have an issue with I2C. The Arduino IDE will operate with the latest stable version, but take caution that PlatformIO (as of this update) is still using 1.0.6.

These three application notes from Freescale Semiconductor (manufacturer of the MPR121 chip) are the most relevant for this use case.

AN3889, MPR121 Capacitance Sensing Settings - Application Notes (nxp.com)

AN3890, MPR121 Capacitance Sensing - Filtering and Timing (nxp.com)

AN3863, Designing Touch Sensing Electrodes - Application Notes (nxp.com)

Adafruit's Tutorials provide a good entry point to begin understanding their products

Overview | Adafruit MPR121 12-Key Capacitive Touch Sensor Breakout Tutorial

Overview | Adafruit HUZZAH32 - ESP32 Feather Tutorial

There is no ESPRESSIF or ADAFRUIT implementation of a BLE Keyboard on ESP32 for Arduino IDE, but T-vK's development is extremely useful and stable

T-vK/ESP32-BLE-Keyboard (github.com)

Espressif Arduino Core for ESP32

Getting Started — Arduino-ESP32 2.0.0 documentation (espressif.com)

Some of the items I'm exploring to add.

Circuit Diagram

RPM Letterboard Code

//---Definitions for Capacitive Touch Sensors
 #include <Wire.h>#include <Adafruit_MPR121.h>#ifndef _BV#define _BV(bit) (1 << (bit))#endifAdafruit_MPR121 TS1 = Adafruit_MPR121();Adafruit_MPR121 TS2 = Adafruit_MPR121();Adafruit_MPR121 TS3 = Adafruit_MPR121();uint16_t lastT1, currT1 = 0;uint16_t lastT2, currT2 = 0;uint16_t lastT3, currT3 = 0;int capT, senT = 0; //--capT=> Touch Sensor Modules (0,1,2), senT=> Individual sensors(0 to 11)bool CapsResetFlag = false;//---Definitions for BLE Keyboard
 #define USE_NIMBLE#include <BleKeyboard.h>BleKeyboard bleKeyboard("RPM Letterboard V2.0");bool keybON, action = false;char L[12][3]; //--adding one for null char in char array
 //---Other Definitions
 int counter = 0;void loadKeyboardMap() { L[2][0] = 'a'; L[3][0] = 'b'; L[2][2] = 'c'; L[3][2] = 'd'; L[2][1] = 'e'; L[3][1] = '1'; L[1][0] = 'f'; L[4][0] = 'g'; L[1][2] = 'h'; L[4][2] = 'i'; L[1][1] = 'j'; L[4][1] = '2'; L[0][0] = 'k'; L[5][0] = 'l'; L[0][2] = 'm'; L[5][2] = 'n'; L[0][1] = ';'; L[5][1] = '3'; L[11][0] = 'o'; L[6][0] = 'p'; L[11][2] = 'q'; L[6][2] = 'r'; L[11][1] = 's'; L[6][1] = '4'; L[10][0] = 't'; L[7][0] = 'u'; L[10][2] = 'v'; L[7][2] = 'w'; L[10][1] = 'x'; L[7][1] = '5'; L[9][0] = 'y'; L[8][0] = 'z'; L[8][2] = 32; L[9][1] = 8; L[8][1] = 10;}void Buzz(int BuzzType) { switch (BuzzType) { case 1: ledcWriteTone(0,1000); delay(100); ledcWrite(0,0); break; //--TS1 is up or scheduled TS reset run
  case 2: ledcWriteTone(0,1500); delay(100); ledcWrite(0,0); break; //--TS2 is up
  case 3: ledcWriteTone(0,2000); delay(100); ledcWrite(0,0); break; //--TS3 is up
  case 4: ledcWriteTone(0,1100); delay(10); ledcWrite(0,0); delay(1000); digitalWrite(A12, HIGH); ledcWriteTone(0,400); delay(10); ledcWrite(0,0); delay(1000); break; //--waiting on BLE pair
  case 5: ledcWriteTone(0,900); delay(300); ledcWriteTone(0,400); delay(700); ledcWrite(0,0); break; //--BLE pairing achieved
  case 6: digitalWrite(A12,HIGH); ledcWriteTone(0,100); delay(2000); ledcWrite(0,0); break; //--fatal error, must reboot manually
  case 7: ledcWriteTone(0,100); delay(300); ledcWriteTone(0,50); delay(700); ledcWrite(0,0); break; //--error, autorestarting TSs
  case 8: digitalWrite(A12,HIGH); ledcWriteTone(0,1500); delay(40); ledcWrite(0,0); digitalWrite(A12,LOW); break; //--low battery
  }}void ResetSensors() { TS1.begin(0x5A); TS2.begin(0x5B); TS3.begin(0x5C); TS1.setThresholds(0x19,0x05); TS2.setThresholds(0x19,0x05); TS3.setThresholds(0x19,0x05);}void ChkResetCapSensors() { int UpTime = int(millis()/1000/60); //--uptime in minutes
  int MinFactor = UpTime % 7; //--divisor is number of minutes between TSs resets
  if (MinFactor != 3) {CapsResetFlag = false;} //--first reset at compared minute
  else if (!CapsResetFlag) { ResetSensors(); // Buzz(1); CapsResetFlag = true;}}void ChkBattery() { float battRead = analogRead(A13); //--or analogRead(35) battRead = ((battRead*2) / 4095 * 3.3 * 1.096); if (battRead < 3.3) { //3.3v as base for beep
  ++counter; if (counter > 80) { Buzz(8); counter = 0; }}}void setup() { //Serial.begin(115200); pinMode(4, OUTPUT); //--ledc for Buzzer : Arduino TONE not implemented for ESP32
  ledcSetup(0, 100000, 12); //--Parms=(channel, freq, resolution) ledcAttachPin(4, 0); pinMode(A12, OUTPUT); //--Built in RED LED
  bleKeyboard.begin(); Wire.begin (23, 22); if (!TS1.begin(0x5A)) {Buzz(6); while (1);} else {Buzz(1);} if (!TS2.begin(0x5B)) {Buzz(6); while (1);} else {Buzz(2);} if (!TS3.begin(0x5C)) {Buzz(6); while (1);} else {Buzz(3);} TS1.setThresholds(0x19,0x05); TS2.setThresholds(0x19,0x05); TS3.setThresholds(0x19,0x05); loadKeyboardMap(); delay(500);}void loop() { while (!bleKeyboard.isConnected()) { keybON = false; digitalWrite(A12, LOW); Buzz(4); } if (bleKeyboard.isConnected() && !keybON) { keybON = true; Buzz(5); digitalWrite(A12, LOW); } ChkResetCapSensors(); ChkBattery(); currT1 = TS1.touched(); currT2 = TS2.touched(); currT3 = TS3.touched(); //--Get the currently touched sensor pads
  for (int i = 0; i < 12; i++) { if ((TS1.filteredData(i)>10000) or (TS2.filteredData(i)>10000) or (TS3.filteredData(i)>10000)) { Buzz(7); ResetSensors(); } //--noise on I2C channel: must reset TSs
  if ((currT1 & _BV(i)) && !(lastT1 & _BV(i))) {capT = 0; senT = i; action = true; } if ((currT2 & _BV(i)) && !(lastT2 & _BV(i))) {capT = 1; senT = i; action = true; } if ((currT3 & _BV(i)) && !(lastT3 & _BV(i))) {capT = 2; senT = i; action = true; }} lastT1 = currT1; lastT2 = currT2; lastT3 = currT3; //--reset TSs states
  if (action) { action = false; switch (L[senT][capT]) { case '1':bleKeyboard.print("si "); break; case '2':bleKeyboard.print("no "); break; case '3':bleKeyboard.print("verdad "); break; case '4':bleKeyboard.print("falso "); break; case '5':bleKeyboard.print("."); break; default :bleKeyboard.print(L[senT][capT]); break; }}}

-----

This article was originally published on Hackster.io.

Open Minds Silicon Valley provides platforms to elevate the voices of diverse students, professionals, and families. We encourage writing submissions to be emailed to eric@openmindschool.org. We look forward to being in touch about possible feature options.