I am running an application on a single-core RT target (PXIe-8101) which does some data acquisition and instrument control. There are 2 - 3 timed loops running, more parallel while loops for data processing and it sends the acquired data to a PC using network streams.
What I observe now is that the load without a client connected (still everything running) is ~ 50% (timed loops 35%, ISRs 6%, other 6%). Upon connection of the client, the load increases a bit to ~ 60% (timed loops 36%, ISRs 8%, other 16%). Things are good if no client is connected. When connecting and starting to receive data, after ~ 5 min the RT target „crashes“, i.e. the ISR load goes to 97%, total load to 100% and the connection to the target is lost from the PC. Only restarting the target helps.
From the timed loops, one is running constantly with priority 50000 @ 1kHz one is triggered by a sample clock @ 1kHz and two are only running sometimes to steer an instrument in given steps @ 200 Hz with priority 60000. In my tests, I never ran the latter two (since they require user interaction).
The RT error log doesn't show anything under "RTLog" or "Exception Log".
So my question is: What could cause the RT target to „crash“ in the described way? What would be good approaches to debug this? Could this have something to do with the network streams? Might connecting the RT target directly to the PC via ethernet instead of going through the building network help?
Any advice is appreciated.