mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-05-05 09:09:13 -05:00
483 lines
17 KiB
Plaintext
483 lines
17 KiB
Plaintext
# Setup Nicla Vision {.unnumbered}
|
|
|
|
The [Arduino Nicla
|
|
Vision](https://docs.arduino.cc/hardware/nicla-vision) (sometimes called
|
|
*NiclaV*) is a development board that includes two processors that can
|
|
run tasks in parallel. It is part of a family of development boards with
|
|
the same form factor but designed for specific tasks, such as the [Nicla
|
|
Sense
|
|
ME](https://www.bosch-sensortec.com/software-tools/tools/arduino-nicla-sense-me/)
|
|
and the [Nicla
|
|
Voice](https://store-usa.arduino.cc/products/nicla-voice?_gl=1*l3abc6*_ga*MTQ3NzE4Mjk4Mi4xNjQwMDIwOTk5*_ga_NEXN8H46L5*MTY5NjM0Mzk1My4xMDIuMS4xNjk2MzQ0MjQ1LjAuMC4w).
|
|
The *Niclas* can efficiently run processes created with TensorFlow™
|
|
Lite. For example, one of the cores of the NiclaV computing a computer
|
|
vision algorithm on the fly (inference), while the other leads with
|
|
low-level operations like controlling a motor and communicating or
|
|
acting as a user interface.
|
|
|
|
> *The onboard wireless module allows the management of WiFi and
|
|
> Bluetooth Low Energy (BLE) connectivity simultaneously.*
|
|
|
|
{width="6.5in"
|
|
height="3.861111111111111in"}
|
|
|
|
## Two Parallel Cores
|
|
|
|
The central processor is the dual-core
|
|
[STM32H747,](https://content.arduino.cc/assets/Arduino-Portenta-H7_Datasheet_stm32h747xi.pdf?_gl=1*6quciu*_ga*MTQ3NzE4Mjk4Mi4xNjQwMDIwOTk5*_ga_NEXN8H46L5*MTY0NzQ0NTg1My4xMS4xLjE2NDc0NDYzMzkuMA..)
|
|
including a Cortex® M7 at 480 MHz and a Cortex® M4 at 240 MHz. The two
|
|
cores communicate via a Remote Procedure Call mechanism that seamlessly
|
|
allows calling functions on the other processor. Both processors share
|
|
all the on-chip peripherals and can run:
|
|
|
|
- Arduino sketches on top of the Arm® Mbed™ OS
|
|
|
|
- Native Mbed™ applications
|
|
|
|
- MicroPython / JavaScript via an interpreter
|
|
|
|
- TensorFlow™ Lite
|
|
|
|
{width="5.78125in"
|
|
height="5.78125in"}
|
|
|
|
## Memory
|
|
|
|
Memory is crucial for embedded machine learning projects. The NiclaV
|
|
board can host up to 16 MB of QSPI Flash for storage. However, it is
|
|
essential to consider that the MCU SRAM is the one to be used with
|
|
machine learning inferences; the STM32H747 is only 1MB, shared by both
|
|
processors. This MCU also has incorporated 2MB of FLASH, mainly for code
|
|
storage.
|
|
|
|
## Sensors
|
|
|
|
- **Camera**: A GC2145 2 MP Color CMOS Camera.
|
|
|
|
- **Microphone**: A
|
|
> [MP34DT05,](https://content.arduino.cc/assets/Nano_BLE_Sense_mp34dt05-a.pdf?_gl=1*12fxus9*_ga*MTQ3NzE4Mjk4Mi4xNjQwMDIwOTk5*_ga_NEXN8H46L5*MTY0NzQ0NTg1My4xMS4xLjE2NDc0NDc3NzMuMA..)
|
|
> an ultra-compact, low-power, omnidirectional, digital MEMS
|
|
> microphone built with a capacitive sensing element and an IC
|
|
> interface.
|
|
|
|
- **6-Axis IMU**: 3D gyroscope and 3D accelerometer data from the
|
|
> LSM6DSOX 6-axis IMU.
|
|
|
|
- **Time of Flight Sensor**: The VL53L1CBV0FY Time-of-Flight sensor
|
|
> adds accurate and low power-ranging capabilities to the Nicla
|
|
> Vision. The invisible near-infrared VCSEL laser (including the
|
|
> analog driver) is encapsulated with receiving optics in an
|
|
> all-in-one small module below the camera.
|
|
|
|
### **HW Installation (Arduino IDE)**
|
|
|
|
Start connecting the board (USB-C) to your computer :
|
|
|
|
{width="6.5in"
|
|
height="3.0833333333333335in"}
|
|
|
|
Install the Mbed OS core for Nicla boards in the Arduino IDE. Having the
|
|
IDE open, navigate to Tools \> Board \> Board Manager, look for Arduino
|
|
Nicla Vision on the search window, and install the board.
|
|
|
|
{width="6.5in"
|
|
height="2.7083333333333335in"}
|
|
|
|
Next, go to Tools \> Board \> Arduino Mbed OS Nicla Boards and select
|
|
Arduino Nicla Vision. Having your board connected to the USB, you should
|
|
see the Nicla on Port and select it.
|
|
|
|
> *Open the Blink sketch on Examples/Basic and run it using the IDE
|
|
> Upload button. You should see the Built-in LED (green RGB) blinking,
|
|
> which means the Nicla board is correctly installed and functional!*
|
|
|
|
## Testing the Microphone
|
|
|
|
On Arduino IDE, go to Examples \> PDM \> PDMSerialPlotter, open and run
|
|
the sketch. Open the Plotter and see the audio representation from the
|
|
microphone:
|
|
|
|
{width="6.5in"
|
|
height="4.361111111111111in"}
|
|
|
|
> *Vary the frequency of the sound you generate and confirm that the mic
|
|
> is working correctly.*
|
|
|
|
## Testing the IMU
|
|
|
|
Before testing the IMU, it will be necessary to install the LSM6DSOX
|
|
library. For that, go to Library Manager and look for LSM6DSOX. Install
|
|
the library provided by Arduino:
|
|
|
|
{width="6.5in"
|
|
height="2.4027777777777777in"}
|
|
|
|
Next, go to Examples \> Arduino_LSM6DSOX \> SimpleAccelerometer and run
|
|
the accelerometer test (you can also run Gyro and board temperature):
|
|
|
|
{width="6.5in"
|
|
height="4.361111111111111in"}
|
|
|
|
### **Testing the ToF (Time of Flight) Sensor**
|
|
|
|
As we did with IMU, installing the ToF library, the VL53L1X is
|
|
necessary. For that, go to Library Manager and look for VL53L1X. Install
|
|
the library provided by Pololu:
|
|
|
|
{width="6.5in"
|
|
height="2.4583333333333335in"}
|
|
|
|
Next, run the sketch
|
|
[proximity_detection.ino](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/distance_image_meter.py):
|
|
|
|
{width="4.947916666666667in"
|
|
height="4.635416666666667in"}
|
|
|
|
On the Serial Monitor, you will see the distance from the camera and an
|
|
object in front of it (max of 4m).
|
|
|
|
{width="6.5in"
|
|
height="4.847222222222222in"}
|
|
|
|
## Testing the Camera
|
|
|
|
We can also test the camera using, for example, the code provided on
|
|
Examples \> Camera \> CameraCaptureRawBytes. We can not see the image
|
|
directly, but it is possible to get the raw image data generated by the
|
|
camera.
|
|
|
|
Anyway, the best test with the camera is to see a live image. For that,
|
|
we will use another IDE, the OpenMV.
|
|
|
|
## Installing the OpenMV IDE
|
|
|
|
OpenMV IDE is the premier integrated development environment for use
|
|
with OpenMV Cameras and the one on the Portenta. It features a powerful
|
|
text editor, debug terminal, and frame buffer viewer with a histogram
|
|
display. We will use MicroPython to program the camera.
|
|
|
|
Go to the [OpenMV IDE page](https://openmv.io/pages/download), download
|
|
the correct version for your Operating System, and follow the
|
|
instructions for its installation on your computer.
|
|
|
|
{width="6.5in"
|
|
height="4.791666666666667in"}
|
|
|
|
The IDE should open, defaulting the helloworld_1.py code on its Code
|
|
Area. If not, you can open it from Files \> Examples \> HelloWord \>
|
|
helloword.py
|
|
|
|
{width="6.5in"
|
|
height="4.444444444444445in"}
|
|
|
|
Any messages sent through a serial connection (using print() or error
|
|
messages) will be displayed on the **Serial Terminal** during run time.
|
|
The image captured by a camera will be displayed in the **Camera
|
|
Viewer** Area (or Frame Buffer) and in the Histogram area, immediately
|
|
below the Camera Viewer.
|
|
|
|
OpenMV IDE is the premier integrated development environment with OpenMV
|
|
Cameras and the Arduino Pro boards. It features a powerful text editor,
|
|
debug terminal, and frame buffer viewer with a histogram display. We
|
|
will use MicroPython to program the Nicla Vision.
|
|
|
|
> *Before connecting the Nicla to the OpenMV IDE, ensure you have the
|
|
> latest bootloader version. To that, go to your Arduino IDE, select the
|
|
> Nicla board, and open the sketch on Examples \> STM_32H747_System
|
|
> STM_32H747_updateBootloader. Upload the code to your board. The Serial
|
|
> Monitor will guide you.*
|
|
|
|
After updating the bootloader, put the Nicla Vision in bootloader mode
|
|
by double-pressing the reset button on the board. The built-in green LED
|
|
will start fading in and out. Now return to the OpenMV IDE and click on
|
|
the connect icon (Left ToolBar):
|
|
|
|
{width="4.010416666666667in"
|
|
height="1.0520833333333333in"}
|
|
|
|
A pop-up will tell you that a board in DFU mode was detected and ask you
|
|
how you would like to proceed. First, select \"Install the latest
|
|
release firmware.\" This action will install the latest OpenMV firmware
|
|
on the Nicla Vision.
|
|
|
|
{width="6.5in"
|
|
height="2.6805555555555554in"}
|
|
|
|
You can leave the option of erasing the internal file system unselected
|
|
and click \[OK\].
|
|
|
|
Nicla\'s green LED will start flashing while the OpenMV firmware is
|
|
uploaded to the board, and a terminal window will then open, showing the
|
|
flashing progress.
|
|
|
|
{width="4.854166666666667in"
|
|
height="3.5416666666666665in"}
|
|
|
|
Wait until the green LED stops flashing and fading. When the process
|
|
ends, you will see a message saying, \"DFU firmware update complete!\".
|
|
Press \[OK\].
|
|
|
|
{width="3.875in"
|
|
height="5.708333333333333in"}
|
|
|
|
A green play button appears when the Nicla Vison connects to the Tool
|
|
Bar.
|
|
|
|
{width="4.791666666666667in"
|
|
height="1.4791666666666667in"}
|
|
|
|
Also, note that a drive named "NO NAME" will appear on your computer.:
|
|
|
|
{width="6.447916666666667in"
|
|
height="2.4166666666666665in"}
|
|
|
|
Every time you press the \[RESET\] button on the board, it automatically
|
|
executes the main.py script stored on it. You can load the
|
|
[main.py](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/main.py)
|
|
code on the IDE (File \> Open File\...).
|
|
|
|
{width="4.239583333333333in"
|
|
height="3.8229166666666665in"}
|
|
|
|
> *This code is the \"Blink\" code, confirming that the HW is OK.*
|
|
|
|
For testing the camera, let\'s run helloword_1.py. For that, select the
|
|
script on File \> Examples \> HelloWorld \> helloword.py,
|
|
|
|
When clicking the green play button, the MicroPython script
|
|
(hellowolrd.py) on the Code Area will be uploaded and run on the Nicla
|
|
Vision. On-Camera Viewer, you will start to see the video streaming. The
|
|
Serial Monitor will show us the FPS (Frames per second), which should be
|
|
around 14fps.
|
|
|
|
{width="6.5in"
|
|
height="3.9722222222222223in"}
|
|
|
|
Let\'s go through the [helloworld.py](http://helloworld.py/) script:
|
|
|
|
```python
|
|
# Hello World Example 2
|
|
#
|
|
# Welcome to the OpenMV IDE! Click on the green run arrow button below to run the script!
|
|
|
|
import sensor, image, time
|
|
|
|
sensor.reset() # Reset and initialize the sensor.
|
|
sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE)
|
|
sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240)
|
|
sensor.skip_frames(time = 2000) # Wait for settings take effect.
|
|
clock = time.clock() # Create a clock object to track the FPS.
|
|
|
|
while(True):
|
|
clock.tick() # Update the FPS clock.
|
|
img = sensor.snapshot() # Take a picture and return the image.
|
|
print(clock.fps())
|
|
```
|
|
|
|
|
|
In GitHub, you can find the Python scripts used here.
|
|
|
|
The code can be split into two parts:
|
|
|
|
- **Setup**: Where the libraries are imported and initialized, and the
|
|
> variables are defined and initiated.
|
|
|
|
- **Loop**: (while loop) part of the code that runs continually. The
|
|
> image (img variable) is captured (a frame). Each of those frames
|
|
> can be used for inference in Machine Learning Applications.
|
|
|
|
To interrupt the program execution, press the red \[X\] button.
|
|
|
|
> *Note: OpenMV Cam runs about half as fast when connected to the IDE.
|
|
> The FPS should increase once disconnected.*
|
|
|
|
In [[the GitHub, You can find other Python
|
|
scripts]{.underline}](https://github.com/Mjrovai/Arduino_Nicla_Vision/tree/main/Micropython).
|
|
Try to test the onboard sensors.
|
|
|
|
## Connecting the Nicla Vision to Edge Impulse Studio
|
|
|
|
We will use the Edge Impulse Studio later in other exercises. [Edge
|
|
Impulse I](https://www.edgeimpulse.com/)s a leading development platform
|
|
for machine learning on edge devices.
|
|
|
|
Edge Impulse officially supports the Nicla Vision. So, for starting,
|
|
please create a new project on the Studio and connect the Nicla to it.
|
|
For that, follow the steps:
|
|
|
|
- Download the [last EI
|
|
> Firmware](https://cdn.edgeimpulse.com/firmware/arduino-nicla-vision.zip)
|
|
> and unzip it.
|
|
|
|
- Open the zip file on your computer and select the uploader related
|
|
> to your OS:
|
|
|
|
{width="4.416666666666667in"
|
|
height="1.5520833333333333in"}
|
|
|
|
- Put the Nicla-Vision on Boot Mode, pressing the reset button twice.
|
|
|
|
- Execute the specific batch code for your OS for uploading the binary
|
|
> (arduino-nicla-vision.bin) to your board.
|
|
|
|
Go to your project on the Studio, and on the Data Acquisition tab,
|
|
select WebUSB (1). A window will appear; choose the option that shows
|
|
that the Nicla is pared (2) and press \[Connect\] (3).
|
|
|
|
{width="6.5in"
|
|
height="4.319444444444445in"}
|
|
|
|
In the Collect Data section on the Data Acquisition tab, you can choose
|
|
what sensor data you will pick.
|
|
|
|
{width="6.5in"
|
|
height="4.319444444444445in"}
|
|
|
|
For example. IMU data:
|
|
|
|
{width="6.5in"
|
|
height="4.319444444444445in"}
|
|
|
|
Or Image:
|
|
|
|
{width="6.5in"
|
|
height="4.319444444444445in"}
|
|
|
|
And so on. You can also test an external sensor connected to the Nicla
|
|
ADC (pin 0) and the other onboard sensors, such as the microphone and
|
|
the ToF.
|
|
|
|
### **Expanding the Nicla Vision Board (optional)**
|
|
|
|
A last item to be explored is that sometimes, during prototyping, it is
|
|
essential to experiment with external sensors and devices, and an
|
|
excellent expansion to the Nicla is the [Arduino MKR Connector Carrier
|
|
(Grove
|
|
compatible)](https://store-usa.arduino.cc/products/arduino-mkr-connector-carrier-grove-compatible).
|
|
|
|
The shield has 14 Grove connectors: five single analog inputs, one
|
|
single analog input, five single digital I/Os, one double digital I/O,
|
|
one I2C, and one UART. All connectors are 5V compatible.
|
|
|
|
> *Note that besides all 17 Nicla Vision pins that will be connected to
|
|
> the Shield Groves, some Grove connections are disconnected.*
|
|
|
|
{width="6.5in"
|
|
height="4.875in"}
|
|
|
|
This shield is MKR compatible and can be used with the Nicla Vision and
|
|
the Portenta.
|
|
|
|
{width="4.34375in"
|
|
height="5.78125in"}
|
|
|
|
For example, suppose that on a TinyML project, you want to send
|
|
inference results using a LoRaWan device and add information about local
|
|
luminosity. Besides, with offline operations, a local low-power display
|
|
as an OLED display is advised. This setup can be seen here:
|
|
|
|
{width="6.5in"
|
|
height="4.708333333333333in"}
|
|
|
|
The [Grove Light
|
|
Sensor](https://wiki.seeedstudio.com/Grove-Light_Sensor/) would be
|
|
connected to one of the single Analog pins (A0/PC4), the [LoRaWan
|
|
device](https://wiki.seeedstudio.com/Grove_LoRa_E5_New_Version/) to the
|
|
UART, and the [OLED](https://arduino.cl/producto/display-oled-grove/) to
|
|
the I2C connector.
|
|
|
|
The Nicla Pins 3 (Tx) and 4 (Rx) are connected with the Shield Serial
|
|
connector. The UART communication is used with the LoRaWan device. Here
|
|
is a simple code to use the UART.:
|
|
|
|
```python
|
|
# UART Test - By: marcelo_rovai - Sat Sep 23 2023
|
|
|
|
import time
|
|
from pyb import UART
|
|
from pyb import LED
|
|
|
|
redLED = LED(1) # built-in red LED
|
|
|
|
# Init UART object.
|
|
# Nicla Vision's UART (TX/RX pins) is on "LP1"
|
|
uart = UART("LP1", 9600)
|
|
|
|
while(True):
|
|
uart.write("Hello World!\r\n")
|
|
redLED.toggle()
|
|
time.sleep_ms(1000)
|
|
|
|
```
|
|
|
|
To verify if the UART is working, you should, for example, connect
|
|
another device as an [Arduino
|
|
UNO](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Arduino-IDE/teste_uart_UNO/teste_uart_UNO.ino),
|
|
displaying the Hello Word.
|
|
|
|
{width="2.8125in"
|
|
height="3.75in"}
|
|
|
|
Here is a Hello World code to be used with the I2C OLED. The MicroPython
|
|
SSD1306 OLED driver (ssd1306.py), created by Adafruit, should also be
|
|
uploaded to the Nicla (the
|
|
[[ssd1306.py]{.underline}](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/ssd1306.py)
|
|
can be found in GitHub).
|
|
|
|
|
|
```python
|
|
# Nicla_OLED_Hello_World - By: marcelo_rovai - Sat Sep 30 2023
|
|
|
|
#Save on device: MicroPython SSD1306 OLED driver, I2C and SPI interfaces created by Adafruit
|
|
import ssd1306
|
|
|
|
from machine import I2C
|
|
i2c = I2C(1)
|
|
|
|
oled_width = 128
|
|
oled_height = 64
|
|
oled = ssd1306.SSD1306_I2C(oled_width, oled_height, i2c)
|
|
|
|
oled.text('Hello, World', 10, 10)
|
|
oled.show()
|
|
```
|
|
|
|
Finally, here is a simple script to read the ADC value on pin \"PC4\"
|
|
(Nicla pin A0):
|
|
```python
|
|
|
|
# Light Sensor (A0) - By: marcelo_rovai - Wed Oct 4 2023
|
|
|
|
import pyb
|
|
from time import sleep
|
|
|
|
adc = pyb.ADC(pyb.Pin("PC4")) # create an analog object from a pin
|
|
val = adc.read() # read an analog value
|
|
|
|
while (True):
|
|
|
|
val = adc.read()
|
|
print ("Light={}".format (val))
|
|
sleep (1)
|
|
```
|
|
|
|
The ADC can be used for other valuable sensors, such as
|
|
[Temperature](https://wiki.seeedstudio.com/Grove-Temperature_Sensor_V1.2/).
|
|
|
|
> *Note that the above scripts ([[downloaded from
|
|
> Github]{.underline}](https://github.com/Mjrovai/Arduino_Nicla_Vision/tree/main/Micropython))
|
|
> only introduce how to connect external devices with the Nicla Vision
|
|
> board using MicroPython.*
|
|
|
|
## Conclusion
|
|
|
|
The Arduino Nicla Vision is an excellent *tiny device* for industrial
|
|
and professional uses! However, it is powerful, trustworthy, low power,
|
|
and has suitable sensors for the most common embedded machine learning
|
|
applications such as vision, movement, sensor fusion, and sound.
|
|
|
|
> *On the* *[GitHub
|
|
> repository,](https://github.com/Mjrovai/Arduino_Nicla_Vision/tree/main)
|
|
> you will find the last version of all the codes used or commented on
|
|
> in this exercise.*
|