Debugging

Quick Checks

# Is the LiveKit plugin installed?
gst-inspect-1.0 livekitwebrtcsink

# Can you reach the LiveKit server?
curl -I https://your-livekit-server.com

# Is the camera working?
v4l2-ctl --list-devices
gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! autovideosink

# Is audio working?
arecord -l  # List capture devices
aplay -l    # List playback devices

Common Issues

“livekitwebrtcsink” not found

GStreamer can’t find the LiveKit plugin.

ERROR: No such element or plugin 'livekitwebrtcsink'

Fix: Run the setup script:

./scripts/setup.sh

Or verify it’s in the plugin path:

gst-inspect-1.0 livekitwebrtcsink

“Failed to attach camera”

Camera device doesn’t exist or wrong format.

[MEDIA_PUBLISHER] Failed to attach camera 'head' (/dev/video0)

Check:

# List cameras
v4l2-ctl --list-devices

# Check supported formats
v4l2-ctl -d /dev/video0 --list-formats-ext

Common causes:

  • Wrong device path (try /dev/video1, /dev/video2)

  • Resolution not supported

  • Camera in use by another process

“Failed to create audio GStreamer elements”

ALSA device not found.

Check:

arecord -l  # List capture devices

Fix: Update mic_device in config.yaml:

mic_device: "hw:1,0"  # Try different numbers

Token errors

[MEDIA_PUBLISHER] Failed to connect to LiveKit

Check:

  • Token not expired (check exp claim)

  • Token has correct room name

  • Token has publish permissions

Decode your token at jwt.io to inspect claims.

Regenerate:

./scripts/generate_token.py \
  --api-key YOUR_KEY \
  --api-secret YOUR_SECRET \
  --room robot-room \
  --identity asimov-robot \
  --role robot \
  --ttl 8760

Connection timeout

Can’t reach LiveKit server.

Check:

# Test WebSocket connection
curl -I https://your-livekit-server.com

# Check firewall
sudo ufw status

Common causes:

  • Wrong URL (http vs wss, wrong port)

  • Firewall blocking WebSocket

  • Server not running

Video freezes or stutters

Check bandwidth:

  • 1920x1200 @ 50fps needs ~3-5 Mbps upstream

  • Try lower resolution: 960x600 @ 80fps

Check CPU:

  • VP8 encoding is CPU-intensive

  • On Jetson, use hardware encoding if available

No state data received

Robot state not showing up on operator side.

Check:

  • asimov-firmware running and sending to UDP 8889

  • asimov-manager receiving state (check logs)

  • DataChannel connected (check LiveKit dashboard)

Debug:

# Check if firmware is sending
sudo tcpdump -i lo udp port 8889

# Check asimov-manager logs
./bazel-bin/asimov-manager --debug

Log Levels

./bazel-bin/asimov-manager          # Normal
./bazel-bin/asimov-manager --debug  # Verbose
./bazel-bin/asimov-manager --quiet  # Errors only

GStreamer Debugging

# Enable GStreamer debug output
GST_DEBUG=3 ./bazel-bin/asimov-manager

# More verbose
GST_DEBUG=4 ./bazel-bin/asimov-manager

# Only WebRTC
GST_DEBUG=livekitwebrtcsink:5 ./bazel-bin/asimov-manager

Checking LiveKit Room

Use livekit-cli to see what’s connected:

livekit-cli list-rooms \
  --url https://your-server.com \
  --api-key YOUR_KEY \
  --api-secret YOUR_SECRET

livekit-cli list-participants \
  --url https://your-server.com \
  --api-key YOUR_KEY \
  --api-secret YOUR_SECRET \
  --room robot-room

Performance

CPU Usage

Component

Typical CPU

Video encode (VP8)

20-40% per stream

Audio encode (Opus)

<5%

Main loop (100 Hz)

<5%

Memory

asimov-manager uses ~50-100 MB RAM. If memory grows over time, check for GStreamer leaks:

valgrind --leak-check=full ./bazel-bin/asimov-manager

Latency

Path

Typical Latency

Camera → LiveKit

50-150 ms

Command → Robot

20-50 ms

State → Operator

20-50 ms

Latency depends on network conditions and WebRTC negotiation.