These six techniques can go a long way in assisting the debugging process:
1. Simplify complex data: Substitute complex analog inputs with digital, synthetic, known, and repeatable data. If a counting pattern does the trick to start, use a counting pattern. For algorithms with coefficients (such as filters), replace the production coefficients with a simpler set of coefficients (such as all zeros except one full-scale coefficient).
2. Divide and conquer: Home in on the location of the problem as much as possible before trying to understand it. This can save significant time both in simulation and in the lab on hardware in the long run. To elaborate, let's say you've got a chained set of processing-intensive signal processing algorithms; let's call them A, B, C, D. When you stimulate A and observe D, you notice a problem. If you can narrow down that the input to C looks fine, but the output of C is bad, you've just homed in on the bug and the problem is likely within C. Now you can strip away dealing with the additional complexities of A, B, and D.
This concept also works from a bottom-up perspective. Verify that the lower level functions work first, then work up through the layers of abstraction. For example, perhaps you've got an analog data logger algorithm. Start by verifying the low level A/D interface, physical memory interface, and physical communication bus interface. Then move up a layer to verify functions like data handlers and routers. Finally, verify the top level of the data logger.
3. Slow the process down: Slow down clocks and data rates. Perhaps some timing requirement isn't being met. Maybe one algorithm isn't synchronized with another. Perhaps one of the algorithms that needs to be data-driven was accidentally coded to be clock-driven. Or maybe one device is trying to push data too quickly to another.
4. Only change one variable at a time: When there are multiple inputs, it can be valuable to isolate them and only change one input at a time, observing the system response. The trick here is to identify all the inputs that induce a change in the output of the system. This is related to fault isolation and activation.
5. Create off-line models: These models should match as closely as possible the real-time algorithm, for bit-level comparison. This is most useful for non-intuitive transformations (such as matrix operations) where it's too challenging to understand what the output should look like relative to the input. If the real-time algorithm is fixed-point and the off-line algorithm is floating-point, there may be small discrepancies that accumulate and cause problems.
6. Start from a known-good state: Slowly tune the input until something breaks, and the bug is observable. This can be helpful for scenarios where overflow may be occurring. For example, if bit wrapping is suspected, slowly increase the amplitude at the input until the problem is observed. Similarly, if there are buffers in use, start by feeding a small amount of data into the algorithm and slowly increasing until the suspected condition occurs. Alternatively, maybe a control loop that breaks with large transients may appear fine with smaller deviations around a steady-state condition.