3d Camera Build Attempt (Embedded Kinect?)

I’ve done a project in the past where I was able to control lights in my house by just pointing at them.

Link Here: https://www.instagram.com/p/BnSZsLSnd-S/?utm_source=ig_web_copy_link

This is achieved by using 3D time of flight cameras like the Xbox Kinect, or Intel Realsense If you’re fancy. The major limitation I’ve encountered is that I’m unable to use either in an embedded environment. I could only use them when plugged into a computer. The Intel Realsense Camera is supposedly made specifically for embedded devices but it uses MIPI protocol which is proprietary and USB 3.0 which if your device supports that it might as well be a damn computer. So with those options out of the way the question remained, how can I give 3d vision to an embedded microcontroller?. That’s the problem I’m attempting to solve. Can’t guarantee that I’ll be able to solve it but I can guarantee that I’ll post my findings to this thread as I go along. Hopefully with more eyes looking at the problem we’ll be able to find a solution.

TL;DR - I’ll use the STM32H7 microcontroller along with TI’s OPT8241 and OPT9221

First thing is deciding on the major components I know I’ll have to have. I need a microcontroller and a 3D Time of Flight (Tof) Image sensor.

I already know the STM32H7 microcontroller will be good for this job since I’ve used it before so I’ll go with that. The massive 3,300 page datasheet for the STM32H7 can be found here but don’t let the page count intimidate you. What you’ll find is that low page count datasheets are MUCH more scary. The more information the better.

For the Image sensor I just googled “3d time of flight Image sensor” and after a very brief search came across 2 solutions from Texas Instruments:

Solution 1: The OPT8320 and it’s Datasheet

This device is awesome because it has a Time of Flight controller built in that handles all of the neat little details associated with interfacing with the Image sensor. From what I can tell it would just be a matter of configuring the controller (which is a thing in itself) and then just reading off pixels from the parallel interface. Unfortunately life is never that simple. The deal breaker here is the extremely small pixel count 80x60. We can’t do a damn thing with that.

Solution 2: The OPT8241 and it’s Datasheet along with the external controller OPT9221 and It’s Datasheet

This solution is a tad bit more complex. It requires the use of an external controller which will also need it’s own supporting hardware components. This solution gives us QVGA 320x240 resolution which is the best I’ve been able to find accessible to the hobbyist. These devices are all available at Digikey and Mouser. It goes without saying that you want to make sure you can actually buy your parts before you put effort in design. Also, I’ve made sure the parts are still in production by the manufacturer.

The OPT8241 Datasheet gives you plenty of valuable information but hardly any of the nitty gritty details. It’s apparent TI wants us to use their controller instead of driving the sensor ourselves. I’m cool with that.

Looking at page 14 of the OPT8241 datasheet we get an abstract block diagram of an implementation of the two devices. Where I’ve labeled “US” is where our microcontroller will connect. If we do the hardware and software config right we can control everything from that point and not have to worry about the science behind the imaging(ADC Gain, Quantization and all that jazz). The OPT9221 will do that for us.

The 2 main takeaways from this diagram is that the OPT9221 will need DDR SDRAM and The sensor requires an illumination element. I’d bet the RAM serves as a frame buffer. The Illumination element can be thought of as an infrared flash. The sensor will send a predetermined modulated pulse and it can determines distance based on what that pulse “looks” like when it reflects back. Simple enough.

Moving 1 level of abstraction down we see a block diagram of the sensor. Again, It is worth noting that TI has not provided much detail on how to drive this device, instead we are encouraged to use the OPT9221 as a middle man. Still, it’s important to understand what signals are doing what because this is far from a plug and chug black box.

The pin descriptions are as follows (Ripped Straight from the Datasheet Starting Pg. 4):
Columns are as follows: Name__Pin Number__In/Out__I/O Bank__Function

The Illumination signals are responsible for providing a modulated pulse to the illumination element.

The Frame sync signals pulse to identify the start/end of a new frame. A frame being one complete scan of the sensor or one complete image

The Horizontal sync pin signals the start and end of new lines of data within a frame. The amount of times the Horizontal sync pin pulses in a frame will be equal to the height of your frame. A 320x240 image will have 240 Horizontal sync pulses.

There are 16 CMOS data pins. Every time the CLKOUT pin pulses it will trigger an update of the data pins that will represent the next pixel in the frame. For example, imagine a 100x100 picture. That’s 10,000 pixels. The CLKOUT will pulse, then we read the value of the 16 data pins and that value represents pixel [0,0]. The CLKOUT pulses again and we read the data pins again. That value represents pixel[0,1]. The next pulse gives us [0,2] and so on. 16 bits = 2 bytes so we get 2 bytes of data per pulse.

LVDS are Low Voltage Differential Signals that could be used as a serial alternative to the 16 bit CMOS Parallel interface. To hell with that though, I’m sticking with parallel which I already know. I’ll look into LVDS for educational purposes,

I2C pins are industry standard I2C protocol pins. SDA - Serial Data, and SCL Serial Clock. These will be connected to the OPT9221

MCLK Is the clock input that must be provided to the sensor. It is of critical importance that this clock signal be clean and consistent. Here we see the clock needs to be between 12Mhz and 50Mhz and maintain a duty cycle from 48% to 52%. I’m assuming the clock source will either be provided through an external crystal oscillator or from the OPT9221 directly. We’ll see when we get to that.

Here is the pin Footprint for the OPT8241 Sensor. There are 78 pins in total but in our breakdown of the functional block diagram I’m not sure if we’ve even considered half of those. That’s good news. That means the rest of those pins support functionality rather than carry out functionality. They’ll either be connected to ground with or without resistors and caps, connected to positive voltage with or without resistors and caps, or just left completely unconnected. Straightforward so far.

Looking at the datasheet not of the OPT9221 It’s a beast of a controller. It comes in a BGA package with 256 pins. I’m not even going to attempt to solder that myself, a Chinese PCBA factory will do.

The OPT9221 Functional Block Diagram is shown on page 14 of the datasheet.


Starting at the top left and moving counter clockwise we have:

Temperature Compensation - Feedback from the Illumination element for temp compensation

To OPT8241 - Synchronization, Clock pulse, reset and I2C communication to the OPT8241. I2C is too slow for high data throughput so in this case it’s only used to configure settings and transfer small amounts of data.

From OTP8241 - This is where the 16 parallel CMOS data pins connect along with their accompanying clock signal

Optional EEPROM - We’re given the option to store the OPT9221 Firmware in external Serial Peripheral Interface (SPI) Electrically Erasable Programmable Read Only Memory (EEPROM). SPI is a very common protocol that requires 4 Lines. The block diagram labels these TIC_MS for some weird reason which is confusing because that label isn’t use anywhere else in the document. Anyways, the 4 lines are Master In Slave out (MISO), Master Out Slave IN (MOSI), Serial Clock (SCLK), and Slave Select (SS) also called Chip Select (CS) .

To DDR This is where we connect to external RAM. About 43 of those 256 pins will be used here. That’s a relief, it knocks off a sizable chunk. We have to be extremely careful to keep best practices when routing DDR on the PCB

Master I2C I2C lines with the OPT9221 as a master and optional temp sensor as a slave

Slave I2C This is where we come in. The Microcontroller will connect here as a master to configure and control the OPT9221

Flow Control This is where Data and Synchronization signals flow back and forth. We have the option of using Digital Vidoe Port (DVP) or Synchronous Serial Interface. We’ll use DVP. It consists of 3 synchronization lines Vertical Sync, Horizontal Sync, Pixel Clock and a number of data lines. In this case 8 data lines that will be updated with each pulse of the pixel clock. You’ll see these sync pins with different names in different documentation.

GPIOs General Purpose Input Output. We may or may not end up using those. If we don’t use them we’ll have to find out whether we need to tie them High, Low, or leave them floating.

Configuration The OPT9221 has 3 configuration pins that are sampled on power up. based on the state of these 3 pins it knows where it ought to load it’s firmware from. See chart below from page 39 of the datasheet

I may be jumping ahead a little with the next diagram but while we’re at it let’s look at the boot sequence on that same page 39

Very important information here, I’ll walk through the diagram. Looking at the Vcc line we learn that our supply voltage can’t turn on too fast or too slow. T_ramp gives us our tolerance. The supply voltage must take between 50 and 3000 microseconds to turn on, that’s what T_ramp represents.

The internal POR line represents an Internal Power ON Reset where the device clears all data and restores itself to default. T_por represents this time frame. The chart tells us it could take up to 200ms.

After POR it samples the Boot Configuration pins and loads the firmware from the specified path. This could take any amount of time based on the config mode, clock rate, amount of data and so forth. This is what T_conf represents.

The device then takes another 2 ms just to get ready after which it will pull the INT_OUT pin low. INT_OUT is pin No. G2 on the OPT9221. Our microcontroller has to make sure this pin has been driven low before attempting any comunication.

Back to the Block Diagram…

Next on the diagram are 6 separate voltage sources with 4 different voltage values. We will have to use a power management IC to handle all of those different voltages. At this point, manual reflow soldering is out of the question. Too many expensive BGA parts.

SYSCLK_IN An external Clock signal that needs to be either 6, 12, 24, or 48 MHz. (Page 12 of datasheet)

Lastly there’s VD_IN which is just an external synchronization pin used when the OPT9221 Timing Generator is in slave mode. We’ll be using the Timing generator as a master so we won’t worry about this. We need only to know whether to tie it High, Low, or leave it floating.

Of the 256 pins, these functional parts take up less than half maybe, I haven’t counted. Point is there are a bunch of pins either tied High or Low, or left unconnected. A lot to digest here. One remaining question is how exactly will our Illumination element work. I’d imagine the use of a near infrared laser that is diffracted to get a wide viewing angle. I don’t know, will cross that bridge when we get to it.