unicam "hangs" - no CSI line data received?

Question

unicam "hangs" - no CSI line data received?

kralo opened this issue a year ago · comments

Describe the bug

streaming video from an ov9281 camera, but suddenly no more frames arriving.
Unicam stuck at "lines done 400" (out of 800). No more error messages.

When I simply stop the application and open another application, e.g. v4l2-ctl --stream-mmap I get images immediately.

Would there be a timeout if the camera "hang" and would unicam report it?
What can possibly derail the CSI path that would be this instantly recoverable?
Are there any "error stats" or "missed interrupts"/counter for the unicam block?

Steps to reproduce the behaviour

start streaming from ov9281 camera and wait

Device (s)

Raspberry Pi CM4 Lite

System

 $ vcgencmd version
Dec 12 2022 11:56:56
Copyright (c) 2012 Broadcom
version ed6f6b8fcdc6476410b9cf75d141633461d34bdd (clean) (release) (start)

Raspberry Pi reference 2022-07-28

Linux tester 5.15.84-v8+ #1610 SMP PREEMPT Mon Dec 19 18:54:50 GMT 2022 aarch64 GNU/Linux

Logs

( line breaks only for readability. not shortened)

[12779.337926] unicam fe800000.csi: ISR: ISTA: 0x1, STA: 0x4020, sequence 433, lines done 0
[12779.337948] unicam fe800000.csi: Scheduling dummy buffer for node 0
[12779.341276] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4023, sequence 433, lines done 200
[12779.344308] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4021, sequence 433, lines done 400
[12779.347342] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4023, sequence 433, lines done 600
[12779.350379] unicam fe800000.csi: ISR: ISTA: 0x0, STA: 0xC020, sequence 433, lines done 800
[12779.350545] video0: VIDIOC_DQBUF: 03:32:59.116510 index=1, type=vid-cap, request_fd=0, flags=0x00002001, field=any, sequence=433, memory=mmap, bytesused=1024000, offset/userptr=0xfa000, length=1024000
[12779.350577] timecode=00:00:00 type=0, flags=0x00000000, frames=0, userbits=0x00000000
[12779.354959] video0: VIDIOC_QBUF: 00:00:00.000000 index=1, type=vid-cap, request_fd=0, flags=0x00002003, field=any, sequence=0, memory=mmap, bytesused=1024000, offset/userptr=0xfa000, length=1024000
[12779.355000] timecode=00:00:00 type=0, flags=0x00000000, frames=0, userbits=0x00000000

[12779.377921] unicam fe800000.csi: ISR: ISTA: 0x1, STA: 0x4022, sequence 434, lines done 0
[12779.377942] unicam fe800000.csi: Scheduling dummy buffer for node 0
[12779.381271] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4021, sequence 434, lines done 200
[12779.384301] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4023, sequence 434, lines done 400
[12779.387334] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4021, sequence 434, lines done 600
[12779.390375] unicam fe800000.csi: ISR: ISTA: 0x0, STA: 0xC022, sequence 434, lines done 800
[12779.390448] video0: VIDIOC_DQBUF: 03:32:59.156508 index=0, type=vid-cap, request_fd=0, flags=0x00002001, field=any, sequence=434, memory=mmap, bytesused=1024000, offset/userptr=0x0, length=1024000
[12779.390478] timecode=00:00:00 type=0, flags=0x00000000, frames=0, userbits=0x00000000
[12779.395240] video0: VIDIOC_QBUF: 00:00:00.000000 index=0, type=vid-cap, request_fd=0, flags=0x00002003, field=any, sequence=0, memory=mmap, bytesused=1024000, offset/userptr=0x0, length=1024000
[12779.395283] timecode=00:00:00 type=0, flags=0x00000000, frames=0, userbits=0x00000000

[12779.417916] unicam fe800000.csi: ISR: ISTA: 0x1, STA: 0x4020, sequence 435, lines done 0
[12779.417938] unicam fe800000.csi: Scheduling dummy buffer for node 0
[12779.421269] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4023, sequence 435, lines done 200
[12779.424299] unicam fe800000.csi: ISR: ISTA: 0x4, STA: 0x4021, sequence 435, lines done 400
  ( not shortened - but PC unattended )
[13152.787211] videodev: v4l2_open: video0: open (0)
[13152.787248] video0: VIDIOC_QUERYCAP: driver=unicam, card=unicam, bus=platform:fe800000.csi, version=0x00050f54, capabilities=0xa5a00001, device_caps=0x25200001
[13152.787287] video0: VIDIOC_QUERY_EXT_CTRL: error -25: id=0xc0000000, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000, elem_size=0, elems=0, nr_of_dims=0, dims=0,0,0,0
[13152.787328] video0: VIDIOC_TRY_EXT_CTRLS: error -25: which=0x0, count=0, error_idx=0, request_fd=0
[13152.787366] video0: VIDIOC_QUERYCTRL: error -25: id=0x80000000, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[13152.787401] video0: VIDIOC_G_SELECTION: error -25: type=vid-cap, target=0, flags=0x0, wxh=0x0, x,y=0,0
[13152.787435] video0: VIDIOC_QUERYCAP: driver=unicam, card=unicam, bus=platform:fe800000.csi, version=0x00050f54, capabilities=0xa5a00001, device_caps=0x25200001
[13152.787610] video0: VIDIOC_QUERY_EXT_CTRL: error -25: id=0xc0000000, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000, elem_size=0, elems=0, nr_of_dims=0, dims=0,0,0,0
[13152.787652] video0: VIDIOC_QUERYCTRL: error -25: id=0xc0000000, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[13152.787686] video0: VIDIOC_QUERYCTRL: 
[13152.789122] video0: VIDIOC_QUERYCTRL: error -25: id=0x98092a, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[13152.789154] video0: VIDIOC_QUERYCTRL: error -25: id=0x8000000, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[13152.789198] unicam fe800000.csi: =================  START STATUS  =================
[13152.789208] unicam fe800000.csi: -----Receiver status-----
[13152.789215] unicam fe800000.csi: V4L2 width/height:   1280x800
[13152.789223] unicam fe800000.csi: Mediabus format:     00002001
[13152.789230] unicam fe800000.csi: V4L2 format:         59455247
[13152.789237] unicam fe800000.csi: Unpacking/packing:   0 / 0
[13152.789244] unicam fe800000.csi: ----Live data----
[13152.789250] unicam fe800000.csi: Programmed stride:   1280
[13152.789257] unicam fe800000.csi: Detected resolution: 640x800
[13152.789263] unicam fe800000.csi: Write pointer:       db87df00
[13152.789270] unicam fe800000.csi: ==================  END STATUS  ==================
[13152.789276] video0: VIDIOC_LOG_STATUS
[13153.547184] video0: VIDIOC_CROPCAP: error -25: type=vid-cap, bounds wxh=0x0, x,y=0,0, defrect wxh=0x0, x,y=0,0, pixelaspect 0/0
[13153.547237] video0: VIDIOC_CROPCAP: error -25: type=vid-out, bounds wxh=0x0, x,y=0,0, defrect wxh=0x0, x,y=0,0, pixelaspect 0/0
[13153.547288] videodev: v4l2_release: video0: release

Additional context

No response

6by9 · Answer 1 · Fri Jan 06 2023 22:36:22 GMT+0800 (China Standard Time)

OV9281 is not a module sold by Raspberry Pi, therefore support is limited.

When I simply stop the application and open another application, e.g. v4l2-ctl --stream-mmap I get images immediately.

Doing so will have totally reset the OV9281. Any issue within the sensor will have been cleared.

Would there be a timeout if the camera "hang" and would unicam report it?

It's not Unicam's place to time out and do anything as it doesn't know anything about the framerate expected. Some sensors support exposure times of >100seconds.
Libcamera does implement a timeout as it is controlling frame rate.

What can possibly derail the CSI path that would be this instantly recoverable?

Almost certainly this is a glitch in the sensor stopping it transmitting. A long ribbon cable causing brownouts, or failing regulator would do it.

Are there any "error stats" or "missed interrupts"/counter for the unicam block?

The main one to look at would be bit 1 of UNICAM_STA which indicates whether a HS clock is detected or not.
That could be added to log_status, but otherwise you can use sudo memtool md 0xfe801000+4 (assumes Pi4/Pi400/CM4) to dump out that register.
Do note that if the sensor runs in non-continuous clock mode (which OV9281 does), then this will indicate no HS clock during the blanking periods.

6by9 · Answer 2 · Fri Jan 06 2023 23:16:37 GMT+0800 (China Standard Time)

Actually UNICAM_STA bit 1 isn't useful as it doesn't get reset to show the current status, you have to write a 1, then a 0, and then monitor it again.

Bit 0 is the packet framer status and does seem to reflect whether data is being received, but note that it will go low between packets.

UNICAM_CLK (0xfe801010) bits 24-27 give the CSI state for the clock lane
0 = STOP
1 = HS_REQ
2 = HS_TERM
3 = HS_PRPR
4 = ULPS
5 = ULPS_REQ
6 = HS_CLK
7 = HS_END
12 = ULPS_EXIT
13 = ERROR

The same bits 24-27 in UNICAM_DATx give a similar state for each of the data lanes
0 = STOP
1 = HS_REQ
2 = HS_TERM
3 = HS_PRPR
4 = LP_YIELD
5 = LP_REQ
6 = HS_DATA
7 = HS_END
10 = ERROR
12 = ESC_REQ
13 = ESC_GO
14 = ULPS_EXIT
15 = ULPS
These definitely do update live, as I've had to run in the VGA 120fps mode to see them change (the chances at 30fps are pretty low as it drops back to STOP between packets)

And I tell a lie about OV9281 running in non-continuous clock mode. It ignores DT and is actually in continuous clock mode.
6.2 has just switched to the upstream OV9282 driver which also supports OV9281. That does look at DT, which is currently configured for non-continuous clock. That drops the max frame rate achievable very slightly as stopping the clock lane takes time.

kralo · Answer 3 · Sat Jan 07 2023 00:07:12 GMT+0800 (China Standard Time)

Thank you. I understand you don't want to support the sensor, but I think you are the best source for the unicam part.
As with many image related things, there isn't a datasheet that fell off the cliff somewhere?

Dumping the memory in the 'fail' state yields

# memtool md 0xfe801000
fe801000: 00080f12 00000000 00000008 00000004                ................
fe801010: 0d000005 00000602 0a000000 0a000000                ................
fe801020: 0a000002 0a000002 00000602 80000101                ................
fe801030: 00000000 00000000 00000000 00001806                ................
fe801040: 00001806 00001806 00001806 00001806                ................
fe801050: 7563616d 7563616d 7563616d 7563616d                macumacumacumacu

with the notable difference that 0xfe801000 = 0 on a streaming/working sensor and fe801010 is 0d000002. Will try to figure that out.

6by9 · Answer 4 · Sat Jan 07 2023 00:52:03 GMT+0800 (China Standard Time)

There's no public datasheet for Unicam.
https://pdfcoffee.com/ov9281-datasheet-pdf-free.html is an uncontrolled datasheet for OV9281.

When it fails it'd be worth reading back from the sensor whether it believes it is streaming or not.
i2ctransfer -y -f 10 w2@0x60 0x01 0x00 r1@060
0x00 is stopped, 0x01 is streaming.

fe801010: 06000005 means that the clock lane is registering HS_CLK
fe801018 and fe80101c: c0000005 means that data lanes 0 & 1 are in the STOP state.

On my streaming sensor I'm seeing

pi@raspberrypi:~ $ sudo memtool md 0xfe801000
fe801000: 00080f03 00000002 00000000 00000e85                ................
fe801010: 06000005 00000602 c0000005 c0000005                ................
fe801020: 0a000002 0a000002 00000602 80000301                ................
fe801030: 00000000 9d000101 00000000 00001806                ................
fe801040: 00001806 00001806 00001806 00001806                ................

When running at 120fps I'm seeing fe801018 and fe80101c reading either 0x06000005 (6 = HS) or 0xc0000005 (0 = STOP).

If I use i2ctransfer -y -f 10 w3@0x60 0x01 0x00 0x00 to deliberately stop the sensor, libcamera is timing out and restarting it on my behalf. v4l2-ctl just stops dead, and I can restart it later with i2ctransfer -y -f 10 w3@0x60 0x01 0x00 0x01.
Do note that sending random I2C commands to a sensor is not recommended, so test purposes only.

kralo · Answer 5 · Wed Feb 01 2023 22:33:02 GMT+0800 (China Standard Time)

The deliberate sending with i2ctransfer fails, do you have any hint?

# i2ctransfer -y -f 10 w2@0x60 0x01 0x00 r1@060
Error: Sending messages failed: Remote I/O error
# i2cdetect -y 10
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- 0c -- -- --
10: 10 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 2f
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- UU -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: UU -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

6by9 · Answer 6 · Wed Feb 01 2023 22:55:54 GMT+0800 (China Standard Time)

Remote I/O error is error -121 EREMOTEIO. The I2C slave hasn't ack'ed a byte, ie it isn't responding.

Absolutely nothing the Pi can do about that.

If the particular module you have has connected up the camera regulator GPIO to the regulators on the OV9281 board, then potentially toggling the regulator GPIO will bring it back to life, although it'll come back up in a stopped state.
Linux stops you getting to a GPIO that is claimed by a regulator, but you can hack it using the rpi3-gpiovirtbuf binary from https://github.com/raspberrypi/raspiraw/blob/master/rpi3-gpiovirtbuf with:

read GPIO state ./rpi3-gpiovirtbuf g 133
set GPIO /rpi3-gpiovirtbuf s 133 1
clear GPIO /rpi3-gpiovirtbuf s 133 0

There should be nothing else that disables that regulator, but it might be worth monitoring pin 17 of CAM1 (CAM_GPIO) to see if it ever erroneously goes low.