Jellyfin 10.11 performance issues are almost always the inotify watcher limits or a bad transcode path. It’s not the CPU. Here’s the exact fix I use on Proxmox LXC.

Check the inotify limit first
Jellyfin watches your media folders for changes. The default inotify limit on Debian is 8192, and Jellyfin burns through that fast. When it hits the limit, you get stuttering and slow library scans. Check it:
cat /proc/sys/fs/inotify/max_user_watches
If it’s 8192, that’s your problem. Bump it up:
echo 'fs.inotify.max_user_watches=524288' >> /etc/sysctl.conf
sysctl -p
This is the same inotify issue I wrote about before with disabling inotify on Proxmox, but here you don’t want to disable it—just raise the limit.
Transcode path is the other big one
If your CPU is pegged during playback, check the transcode directory. Jellyfin defaults to /var/lib/jellyfin/transcodes, and if that’s on a slow disk or a small tmpfs, you’ll get buffering. I’d move it to a dedicated tmpfs:
mkdir -p /mnt/transcodes
mount -t tmpfs -o size=8G tmpfs /mnt/transcodes
chown jellyfin:jellyfin /mnt/transcodes
Then set the transcode path in Jellyfin’s dashboard to /mnt/transcodes. 8GB is enough for a couple of 4K streams. If you’re low on RAM, use a fast SSD instead.

LXC container tweaks
If you’re running Jellyfin in an unprivileged LXC container, you need to allow the container to use more resources. Add these to the container config:
lxc.cgroup2.memory.max = 8G
lxc.cgroup2.cpu.max = 800000 100000
That gives it 8GB RAM and 8 CPU cores. Adjust to your hardware. Also make sure the container has access to the GPU if you’re using hardware acceleration—that’s a whole other rabbit hole.
What didn’t work
I tried the GitHub issue #15685 suggestions—disabling the dashboard, lowering the library scan interval—and they did nothing. The inotify bump and the transcode path fixed it for me. Your mileage may vary, but start there.