There are a few queues at play here:

* Depending on the CAN device, the CAN device's receive buffer needs to be drained by the kernel before it overflows. In the case of register-based (on-chip) CAN devices this will always happen, because the IRQ will cause the kernel to service the register immediately and copy the data into a socket structure. In the case of SPI based devices like MCP2515, you can have problems at this layer already, because the SPI request to drain the buffer can end up in a kernel workqueue which can get preempted and cause the CAN device's buffer to overflow. Make sure the spi tasks have realtime priority. And in the case of USB or other devices, it's at the mercy of the driver and hardware - most USB <-> CAN devices have their own sizeable internal memory buffer in firmware, but some don't and who knows if the firmware for each is implemented properly... anyway, it doesn't seem like this is your issue though. If it were, candump would probably make things worse (more preemption/contention for draining kernel-level queues) rather than better.

* OK, now the CAN data lives in a Linux packet structure. It then goes into the socket routing code. The socket routing code has a few queues in it which can fill and are subject to all kinds of wild and wacky Linux socket priority "disciplines." Try replacing those with the simplest one, pfifo_fast: `sudo tc qdisc replace dev can0 root pfifo_fast`(you can also try `noqueue` but then your application better be really fast...). I suspect this could be your problem as having another application (candump) draining the kernel queues seems to be helping you out.

* Next, your socket itself has a recv buffer/queue, called rmem. If you want to max it out, you can try  `echo "26214400" > /proc/sys/net/core/rmem_max && echo "26214400" > /proc/sys/net/core/rmem_default` . This makes up for application software that can't poll its open socket object fast enough to receive all data.

Hope this helps. For me, the last two (setting the qdisc to pfifo_fast and increasing rmem) made even full-speed CAN communication with Python (slowest runtime ever) pretty reasonable. It's funny that we can't handle 500kbit data transfer, which should be peanuts, but such is adding too many layers...

No results so far, noqueue actually does work on the bench. Changing rmem parameters (I checked from within my application that it is factual) does not change anything, playing with queue disciplines neither (actually, doing noqueue when on the car makes the whole thing lock up immediately, but that was to be expected). And running candump in parallel keeps helping out when using increased rmem and / or different queue...

EDIT: Not only some other minor tweaks and attempts did not solve the problem, now I also got one instance of failed flashing when running candump did not help...

I have been debugging / looking inside the traffic a bit, and my head now hurts. First of all, I got two single messages in my system logs over the last two days about unexpected echo id, which means that my CAN-USB firmware is sending a USB frame that has a wrong echo field, and that can, as far as I can see for now, only happen if I have a memory leak in my firmware, but I do not see it.

The other thing is, I now see that the flashing freezes after 10 consecutive write CAN frames from PC to the ECU. This I have seen before when developing my firmware, and it almost certainly means that the PC / host does not receive a corresponding echo USB frame. Now, knowing that this is triggered by the other CAN traffic from the car would indicate that the USB is not catching up (my firmware decides to skip the USB frame after some timeout if it can be pushed the host, but it is large) or something in the kernel decides to drop the USB frames. Not sure how running candump helps that, probably drains the USB queue quicker.

Since both observations mention echo frames, they could be related, but knowing my luck it is coincidental and I have two problems at hand.

I agree, or a separate process entirely that runs with high prio and you talk to via IPC... a local unix socket or even domain socket for higher level "commands" will do better than the ridiculous libusb stack... as long as your IPC "API" implementation you design is reasonable and relatively timing insensitive.

Can you set some hardware receive filters in your CAN controller to avoid overflowing its buffers if the CAN to SPI to software path is delayed?

Can you try to shut up the other modules during the flash so they don't give you unwanted traffic?

The sad thing is with 4GHz processors and 256G of memory, the already enormous latency of the USB stack has just completely exploded.

You could just use J2534...

*ducks for cover*