Jetson AGX Xavier Argus Timeouts

KirillT · 2024-09-19T11:50:36.066Z

Hello,

I am currently working on an accelerated Gstreamer-based application that utilizes 8 IMX390 GMSL cameras. The camera streams are tiled into a single H265 video stream. My setup is based on a Jetson AGX Xavier device running JetPack 4.6.2 (L4T r32.7.2).

Despite trying numerous solutions available online, I am consistently facing a camera timeout issue. Below is an example error from the nvargus-daemon:

May 30 17:22:43 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 931
May 30 17:22:43 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 930
May 30 17:22:43 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:22:43 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 932
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 931
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 930
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 893)
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 406)
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 506)
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/CameraDriver.cpp, function deleteSession(), line 635)
May 30 17:22:48 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 194)
May 30 17:22:53 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 774
May 30 17:22:53 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 773
May 30 17:22:53 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 772
May 30 17:22:53 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:22:53 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 774
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 773
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 772
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 893)
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 406)
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 506)
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/CameraDriver.cpp, function deleteSession(), line 635)
May 30 17:22:58 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 194)
May 30 17:23:03 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 614
May 30 17:23:03 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 612
May 30 17:23:03 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 613
May 30 17:23:03 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:23:03 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 614
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 612
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 613
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 893)
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 406)
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 506)
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/CameraDriver.cpp, function deleteSession(), line 635)
May 30 17:23:08 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 194)
May 30 17:23:13 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 454
May 30 17:23:13 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 453
May 30 17:23:13 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 452
May 30 17:23:13 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:23:13 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 454
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 453
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 452
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 893)
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 406)
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 506)
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/CameraDriver.cpp, function deleteSession(), line 635)
May 30 17:23:18 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 194)
May 30 17:23:23 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 294
May 30 17:23:23 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 293
May 30 17:23:23 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:23:23 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
May 30 17:23:28 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 294
May 30 17:23:28 47-Jetson nvargus-daemon[5243]: waitForIdleLocked remaining request 293
May 30 17:23:28 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
May 30 17:23:28 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 893)
May 30 17:23:28 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 406)
May 30 17:23:28 47-Jetson nvargus-daemon[5243]: PowerServiceCore:handleRequests: timePassed = 5018
May 30 17:23:29 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 506)
May 30 17:23:29 47-Jetson nvargus-daemon[5243]: SCF: Error Timeout:  (propagating from src/api/CameraDriver.cpp, function deleteSession(), line 635)
May 30 17:23:29 47-Jetson nvargus-daemon[5243]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 194)
May 30 17:23:29 47-Jetson nvargus-daemon[5243]: === python3[32]: WARNING: Cleaning up 8 outstanding streams...(Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 853)

I have explored various approaches to mitigate this problem but have not yet found a successful resolution. Could anyone provide insights or guidance on resolving this issue?

Thank you in advance for your help.

allan.navarro · 2024-09-19T19:41:19.742Z

Hi @KirillT could you try boosting the clocks

sudo jetson_clocks

Also make sure all cores are being used

nvpmodel -m 8

ShaneCCC · 2024-09-20T06:00:43.208Z

If boost the clocks still not fixed have below command to try.

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon&

KirillT · 2024-09-20T10:52:46.625Z

Hi, @allan.navarro . Thank you for the advice. Unfortunately, it did not help. I still see the timeout issue in the logs.

KirillT · 2024-10-08T13:07:24.467Z

Hi, @ShaneCCC . Unfortunately, the infinite timeout also did not help. Providing simplified pipelines

gst-launch-1.0 nvarguscamerasrc sensor-id=0  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

	gst-launch-1.0 nvarguscamerasrc sensor-id=1  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

	gst-launch-1.0 nvarguscamerasrc sensor-id=2  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

	gst-launch-1.0 nvarguscamerasrc sensor-id=3  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

	gst-launch-1.0 nvarguscamerasrc sensor-id=4  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

	gst-launch-1.0 nvarguscamerasrc sensor-id=5  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

	gst-launch-1.0 nvarguscamerasrc sensor-id=6  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

	gst-launch-1.0 nvarguscamerasrc sensor-id=7  bufapi-version=true do-timestamp=true ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720' ! fpsdisplaysink video-sink=fakesink &

I get the following error
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: TIMEOUT

The cameras that fail to capture take longer to finish.

ShaneCCC · 2024-10-09T02:17:52.499Z

Could you verify on 6 cameras.

KirillT · 2024-10-10T06:57:22.062Z

@ShaneCCC The issue is reproducible even with 1 camera.

ShaneCCC · 2024-10-11T03:43:39.442Z

I would suspect it could be sensor driver unstable problem.
How many time to reproduce by one camera?

confirm by v4l2-ctl for the sensor driver stability.
Review the CID function like xxx_set_exposure/xxx_set_frame_rate if v4l2-ctl without problem.

system · 2024-11-06T05:49:58.300Z

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.