# Debugging

## Quick Checks

```bash
# Is the LiveKit plugin installed?
gst-inspect-1.0 livekitwebrtcsink

# Can you reach the LiveKit server?
curl -I https://your-livekit-server.com

# Is the camera working?
v4l2-ctl --list-devices
gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! autovideosink

# Is audio working?
arecord -l  # List capture devices
aplay -l    # List playback devices
```

---

## Common Issues

### "livekitwebrtcsink" not found

GStreamer can't find the LiveKit plugin.

```
ERROR: No such element or plugin 'livekitwebrtcsink'
```

**Fix:** Run the setup script:

```bash
./scripts/setup.sh
```

Or verify it's in the plugin path:

```bash
gst-inspect-1.0 livekitwebrtcsink
```

### "Failed to attach camera"

Camera device doesn't exist or wrong format.

```
[MEDIA_PUBLISHER] Failed to attach camera 'head' (/dev/video0)
```

**Check:**

```bash
# List cameras
v4l2-ctl --list-devices

# Check supported formats
v4l2-ctl -d /dev/video0 --list-formats-ext
```

**Common causes:**
- Wrong device path (try /dev/video1, /dev/video2)
- Resolution not supported
- Camera in use by another process

### "Failed to create audio GStreamer elements"

ALSA device not found.

**Check:**

```bash
arecord -l  # List capture devices
```

**Fix:** Update `mic_device` in config.yaml:

```yaml
mic_device: "hw:1,0"  # Try different numbers
```

### Token errors

```
[MEDIA_PUBLISHER] Failed to connect to LiveKit
```

**Check:**
- Token not expired (check `exp` claim)
- Token has correct room name
- Token has publish permissions

Decode your token at [jwt.io](https://jwt.io) to inspect claims.

**Regenerate:**

```bash
./scripts/generate_token.py \
  --api-key YOUR_KEY \
  --api-secret YOUR_SECRET \
  --room robot-room \
  --identity asimov-robot \
  --role robot \
  --ttl 8760
```

### Connection timeout

Can't reach LiveKit server.

**Check:**

```bash
# Test WebSocket connection
curl -I https://your-livekit-server.com

# Check firewall
sudo ufw status
```

**Common causes:**
- Wrong URL (http vs wss, wrong port)
- Firewall blocking WebSocket
- Server not running

### Video freezes or stutters

**Check bandwidth:**
- 1920x1200 @ 50fps needs ~3-5 Mbps upstream
- Try lower resolution: 960x600 @ 80fps

**Check CPU:**
- VP8 encoding is CPU-intensive
- On Jetson, use hardware encoding if available

### No state data received

Robot state not showing up on operator side.

**Check:**
- asimov-firmware running and sending to UDP 8889
- asimov-manager receiving state (check logs)
- DataChannel connected (check LiveKit dashboard)

**Debug:**

```bash
# Check if firmware is sending
sudo tcpdump -i lo udp port 8889

# Check asimov-manager logs
./bazel-bin/asimov-manager --debug
```

---

## Log Levels

```bash
./bazel-bin/asimov-manager          # Normal
./bazel-bin/asimov-manager --debug  # Verbose
./bazel-bin/asimov-manager --quiet  # Errors only
```

## GStreamer Debugging

```bash
# Enable GStreamer debug output
GST_DEBUG=3 ./bazel-bin/asimov-manager

# More verbose
GST_DEBUG=4 ./bazel-bin/asimov-manager

# Only WebRTC
GST_DEBUG=livekitwebrtcsink:5 ./bazel-bin/asimov-manager
```

## Checking LiveKit Room

Use livekit-cli to see what's connected:

```bash
livekit-cli list-rooms \
  --url https://your-server.com \
  --api-key YOUR_KEY \
  --api-secret YOUR_SECRET

livekit-cli list-participants \
  --url https://your-server.com \
  --api-key YOUR_KEY \
  --api-secret YOUR_SECRET \
  --room robot-room
```

---

## Performance

### CPU Usage

| Component | Typical CPU |
|-----------|-------------|
| Video encode (VP8) | 20-40% per stream |
| Audio encode (Opus) | <5% |
| Main loop (100 Hz) | <5% |

### Memory

asimov-manager uses ~50-100 MB RAM. If memory grows over time, check for GStreamer leaks:

```bash
valgrind --leak-check=full ./bazel-bin/asimov-manager
```

### Latency

| Path | Typical Latency |
|------|-----------------|
| Camera → LiveKit | 50-150 ms |
| Command → Robot | 20-50 ms |
| State → Operator | 20-50 ms |

Latency depends on network conditions and WebRTC negotiation.