Adding a Coral TPU to my NVR (ProxMox LXC)

After getting an IP camera and flashing it with open source firmware it only makes sense if I start recording the feed into some sort of NVR and also expose it to my HomeAssistant.

Browsing around one inevitable lands on Frigate, which is an Open Source NVR. What's cool about it, is that it can use a local model to detect things in images (e.g. cars, people, and amazon packages left on your porch) and it can leverage a Coral TPU to do so.

Coral comes in several flavours, but I've picked an M.2 variant, since I have an open slot in my NAS available. After browsing Amazon, most offerings were to the tune of $150CAD, but looking on Coral's own website, they link you to suppliers, and Mouser sells them for about $40 (+tax/shipping) which is a much better deal.

So, naturally, I've ordered one, and two days later it has arrived. I shut down my NAS and installed it.

After rebooting it was not showing up, which was due to the lack of drivers.

First, I've followed this guide

apt update && apt dist-upgrade -y
apt install wget curl git -y
apt install pve-headers -y
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | tee /etc/apt/sources.list.d/coral-edgetpu.list
mkdir -m 0755 -p /etc/apt/keyrings/
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | tee /etc/apt/keyrings/coral.gpg
apt-get install gasket-dkms libedgetpu1-std -y
sh -c "echo 'SUBSYSTEM==\"apex\", MODE=\"0660\", GROUP=\"apex\"' >> /etc/udev/rules.d/65-apex.rules"
groupadd apex
adduser $USER apex

But that didn't quite work because the version of their deb does not support kernel 6.8.x.

The solution to that was to clone the latest coral driver locally and build it.

apt install pve-headers-$(uname -r)
apt install devscripts dh-dkms
git clone https://github.com/google/gasket-driver.git
cd gasket-driver
debuild -us -uc -tc -b
cd ..
dpkg -i gasket-dkms_1.0-18_all.deb

After rebooting I could see gasket and apex in the output of lsmod and /dev/apex_0 device was present.

Technically it's a dual TPU chip, but I think the M.2 slot I'm using only exposes 1 PCI lane, so I only see one device. Maybe I can play around with the BIOS to switching from 1xPCIx2 to 2xPCIx1, but most likely not, since this slot is reserved for a WiFi card and probably is not even routed fully for this.

One side-effect of this was that my PCI devices numbering got shifted, so I had to adjust my PCI device mappings for the TrueNAS VM from 06:00:00 to 07:00:00

Installing Frigate

For that I've simply used the Frigate Install Helper from Proxmox VE Helper Scripts site

Once installed, we need to pass through the device to the container, which at this point still involves manual editing of the config file.
First, let's establish the device id's, by running ls -la /dev/apex_0

crw-rw---- 1 root apex 120, 0 Jul  4 19:27 /dev/apex_0

Notice the 120, 0 part? Those are the ID's we need to add to the LXC configuration, by editing /etc/pve/lxc/<node_id>.conf. We need to add the following:

lxc.cgroup2.devices.allow: c 120:0 rwm
lxc.mount.entry: /dev/apex_0 dev/apex_0 none bind,optional,create=file

After starting the container I've checked the permissions on the /dev/apex_0 file and saw the following:

root@nas:/etc/pve/lxc# ls -la /dev/apex_0 
crw-rw---- 1 root 1000 120, 0 Jul  4 19:27 /dev/apex_0

1000 is the gid of the file on the host, which corresponds to the apex group. As I understand the driver automatically sets the permissions to that group for any TPU device.

So I've added the apex group inside the container, and also added root and frigate users to it. (Not sure if that's necessary).

Frigate configuration

By default the helper creates a config with CPU detector, which immediately loaded the 12 cores to 70%, and the detection FPS on the sample clip was pretty low...

By going to the config and changing the detector: section to the following I switched it over to the TPU

cameras:
  test:
    enabled: false
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: /media/frigate/person-bicycle-car-detection.mp4
          input_args: -re -stream_loop -1 -fflags +genpts
          roles:
            - detect
            - rtmp
    detect:
      height: 1080
      width: 1920
      fps: 5
detectors:
  coral:
    type: edgetpu
    device: pci
version: 0.14

After saving and restarting, the CPU usage dropped to a couple of percentage points, and all the detection was much faster

The TPU seemingly runs at 50° even when all feeds are disabled, so, I hope my NAS cooling can deal with it. I wonder, if I move it to another M.2 slot and leverage both units if that would increase.

Next thing on the menu is figuring out Frigate and also wiring up the camera to the HomeAssistant.