GDB Tips and Tricks #4: Reverse Debugging

How many times have you stepped through code only to find that you’ve gone too far? Maybe you overstepped a critical point in the code’s execution. Or perhaps you stepped over a function you intended to step into.

Did you know that GDB could actually step backwards through of code? That’s right. You can have GDB go back in time. This is often referred to as “reverse debugging.” But how does it work?

How It Works

Reverse debugging actually relies upon another gem in GDB’s bag-of-tricks called “process record and replay”. Rolls right off the tongue doesn’t it? I won’t spend a lot of time going into the details of PRR here, but it’s quite powerful. The only PRR command we need to be concerned with in this discussion is “record”.

The “record” command begins recording the execution of your application, making notes of things like memory and register values. When you arrive at a point in your application at which you’d like to go backwards, you can issue the “reverse” versions of all the navigation commands you’re already familiar with. This include reverse-step (rs), reverse-next (rn), reverse-continue (rc), and reverse-finish (no short version 🙁 ). As you move backwards through code, gdb reverts the state of memory and registers, effectively unexecuting lines of code.

Let’s see an example using the code snippet below.

#include <iostream>
int sum(int a, int b)
    int result = a + b;
    return result;
int main(int argc, char **argv)
    int a = 12;
    int b = 13;
    int c = sum(a, b);
    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";
    return 0;

Compile this (don’t forget to compile it with the ‘-g’ flag!) and fire up gdb. Then set a breakpoint at main. We can’t begin recording program execution before it’s actually running. So we issue the run command, which will execute our application and promptly break at main.

(gdb) break main
Breakpoint 1 at 0x4007df: file gdbtest.cpp, line 11.
(gdb) run
Starting program: /home/skirk/gdbtest 
Breakpoint 1, main (argc=1, argv=0x7fffffffdf28) at gdbtest.cpp:11
11	    int a = 12;

At this point, we issue the “record” command to begin recording.

(gdb) record

Now let’s start stepping through the code.

(gdb) n
12	    int b = 13;
(gdb) n
13	    int c = sum(a, b);
(gdb) n
15	    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";

We’re now at the point just before the sum is written to stdout. What if I had intended to step into the sum function to see what it’s doing? Let’s back up to just before the sum function is called and then step into it.

(gdb) reverse-next
13	    int c = sum(a, b);
(gdb) s
sum (a=12, b=13) at gdbtest.cpp:5
5	    int result = a + b;

Now we appear to have gone back into time. This allows us to step into the sum function. At this point, we can inspect the values of parameters a and b as we normally would.

(gdb) print a
$1 = 12
(gdb) print b
$2 = 13

If we’re satisfied with the state of things, we can allow the program to continue on.

(gdb) c
No more reverse-execution history.
main (argc=1, argv=0x7fffffffdf28) at gdbtest.cpp:15
15	    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";

An interesting thing happened here. The program execution stopped at the point at which we previously started stepping backwards. When stepping through code using recorded history, “continue” will continue program execution until the history has been exhausted, unless, of course, it has some other reason to stop such as breakpoints and the like.

Let’s now stop the recording process using the “record stop” command and allow the program to continue execution until completion.

(gdb) record stop
Process record is stopped and all execution logs are deleted.
(gdb) c
The sum of 12 and 13 is 25
[Inferior 1 (process 10608) exited normally]

What if we hadn’t stopped recording? Well, it depends. If your version of the runtime executes instructions that aren’t supported by PRR, then you may encounter errors such as this…

Process record does not support instruction 0xc5 at address 0x7ffff7dee8b7.
Process record: failed to record execution log.
Program stopped.
_dl_runtime_resolve_avx () at ../sysdeps/x86_64/dl-trampoline.h:81
81	../sysdeps/x86_64/dl-trampoline.h: No such file or directory.

In this case, AVX instructions are being executed which aren’t supported by the record process. (In this particular case, there’s a workaround. We can export the environment variable LD_BIND_NOW=1 which resolves all symbols at load time. Doing so actually prevents the call to _dl_runtime_resolve_avx later.)

It’s also possible you might see something like…

The sum of 12 and 13 is 25
The next instruction is syscall exit_group.  It will make the program exit.  Do you want to stop the program?([y] or n)

Here you’re prompted as to whether or not you want to stop the program. Regardless of what you choose, you’re still able to navigate backwards in program execution. That’s right – you can reverse debug an application that has finished running.


There are a few caveats when performing reverse debugging.

The first is that you can’t move backwards beyond the point at which you started recording. That should make sense.

Another caveat is that recording isn’t free or cheap. There’s a non-trivial amount of overhead involved in keeping track of registers and memory. So use record where it matters.

By default, there’s an upper limit on the number of instructions the record log can contain. This is 200,000 in the default record mode. This can be tweaked however, including setting it to unlimited (which really just means it’ll record until it’s out of memory). See the GDB manual for more info on this.

You can always see what the current record instruction limit is by using the “info record” command.


Reverse debugging is great tool to keep in your toolbox for those tricky bits of code. In the right contexts, it can save you lots of time. Use it judiciously, however. Recording everything in your application wastes memory, memory that your application may actually need. It can also be detrimental to your program’s execution speed.

DIY Oscilloscope: JYE Tech’s DSO 138

The first time I touched an oscilloscope was in college. I was taking a second semester physics course. And while the course labs made pretty heavy use of oscilloscopes, I, like many of my classmates, learned just enough about them to get by. I’d walk through each lab assignment twisting dials and pushing buttons until I got something on the screen that resembled what I was supposed to see. And that was about it. Any amount of understanding that I actually had fell out of my brain once the semester was over.

As a software professional, the first time I encountered an oscope was while working for a very popular (at the time) mapping software company. The company was developing a new edition of its handheld GPS device and the LCD panel they had been using was officially end-of-lifed. Our hardware folks were having a hard time picking a replacement. It came down to two panels – one from Sharp and one from Epson. I was tasked with implementing a software driver for the Sharp, while someone else got the Epson up and running. For two solid days, I sat in my cube next to an oscope that I really, really hoped I wouldn’t need to use. I spent most of my time banging around on SPI code and reading documentation. However, I’d occasionally hit a wall. That meant putting on my best sad puppy dog face and shuffling as pathetically as I could into one of the hardware guys’ offices to ask for some oscope help. In the end, I managed to get the driver working. But the experience wasn’t great and it bruised my ego somewhat. (As a bonus kick-in-the-fruitbasket, the company opted for the Epson panel.)

I really wish I could say that was an isolated incident. But the truth is that story has more or less repeated itself a few times throughout my career. The companies, colleagues, and products have changed, but the need to use an oscope continues to pop up.

A few months ago, I was watching an old episode of “Know How”. The episode was primarily about a cheap DIY entry-level oscilloscope kit from JYE Tech – the DSO 138. It looked super simple and a lot of fun to play with. There were no built in function generators and no fancy math functions. There were just a few buttons and switches to learn. It also appeared easy to assemble.

As I watched the episode, I relived each of my horrible oscope encounters. I felt slightly embarrassed to have come this far in my career without having some level of comfort with oscopes – even the DSO 138 intimidated me on some level. So I pulled the trigger and bought the kit.

About The DSO 138

As I mentioned above, the DSO 138 is an entry-level, DIY oscilloscope kit from JYE. Let me bold and italicize that phrase one more time – entry-level. It can only handle signals up to 200 KHz. It samples at a maximum rate of 1 million samples per second. And it’s limited to a maximum peak input voltage of 50 V. If your primary use case is audio applications or testing PWM signals, this kit should be just fine for you.

The kit arrives as a bare PCB board along with a 2.4” LCD, a bag of assorted components, test leads, and some assembly instructions. This being a kit, of course, means that you have to solder everything up. And depending on which flavor of the DSO 138 you get, you may or may not have to solder a few surface mount components as well. I’ll have more to say about this in a minute.

The scope is built around the STM32F103C8, which is an ARM-based microcontroller from ST. The STM32F103C8 has a 72MHz CPU, 2 12-bit ADCs (12-bits is the max resolution of the oscope, incidentally), 20KB of SRAM, and 64KB or 128KB of Flash (not sure which version was selected for this kit). It also comes pre-flashed with firmware for this project, so you don’t have to worry about doing it yourself.

The board features two options for power – a traditional barrel connector and a JST connector. There are 3 slide switches – one is for controlling in the input type and the other two are used to select range and sensitivity. Four buttons are used to control various on-screen parameters. A fifth button provides a reset. There’s also a USB connector included in the mix. It’s curious, however, because the USB connector doesn’t actually seem to have a purpose. It’s optional to install and the accompanying documentation says, “It was provided for future or user own use.” I haven’t been brave enough to see if I can power the board from it.

The board also features a built-in 1KHz 3.3V test signal, which you can use to verify that things are working as they should.

Purchasing the DSO 138

You can obtain the DSO 138 from a number of online retailers such as Amazon, Banggood, Ebay, etc. When purchasing the DSO 138, you should probably make sure you buy it from an approved seller. There are a number of counterfeit kits floating around. And JYE doesn’t mince words when it comes to them. They call out sellers by name both on their website and in their firmware splash screen. I purchased mine from an Amazon seller name NooElec, which is apparently on the “approved” list.

For reasons I describe below, I ended up buying two kits at two different times using the same Amazon link. And I got two different flavors of the DSO 138 – the 13803K and the 13804K. The only difference between these two versions are the surface mount components. The 13803K comes with all of the surface mount components presoldered. The 13804K has the ST chip presoldered and nothing else, which means you’ve got more soldering to do.

I thought I had bought the 13803K the first time, but the 13804K showed up on my doorstep. Unless the idea of soldering surface mount resistors strikes fear into your soul, it’s not that big of a deal. Incidentally, this was my first time with SMDs and I had no problem at all.

Something to keep in mind with this kit as well is that it doesn’t come with a case. If you intend to use the DSO 138 on a semiregular basis, you might want to consider purchasing a case. (This is especially true if you use leaded solder when putting it together. :-)) A company named Smartcoco makes cheap clear acrylic cases for the DSO 138 which can be found on Amazon. Unfortunately, it doesn’t come with instructions, so you’ll need to turn to YouTube for help


The DSO 138 assembly process is fairly straightforward. The directions are actually quite good. As in all electronics projects, you should always start out soldering the components with the lowest profiles and then work your way up to the taller, beefier components. It took me approximately three hours to finish soldering the main board (which includes a pee break and at least 2 soda refills).

If you do end up getting the 13804K and you’ve never soldered an SMD before, don’t panic. As I said before, I had never soldered SMD prior to this either. If my fat fingers and shaky hands can do it, so can yours. There are plenty of YouTube videos that provide tips and tricks for this. Just watch a couple of them prior to getting started and you’ll be just fine.

One minor annoyance about this project is that the through-hole resistors are blue (at least they were in my kit). The blue color totally messed with my ability to discern the colors of the resistor’s bands. I had to measure each resistor one by one to figure out what was what. There are only 23 resistors included, so it wasn’t terribly painful. And I suppose you should always measure these things anyway as a way of double-checking. “Measure twice, solder once,” as they say.

I did make the horrible mistake of doing some soldering while tired and feeling rushed. As a result, I soldered the pin connector to the wrong side of the LCD board. Whoops! This, of course, meant that I wasted a LOT of time trying to desolder it without destroying the solder pads or the pin connector…both of which I did anyway.

If, like me, you do screw up the LCD pin connector, you can find a replacement at AccuDIY. However, you’ll probably pay more than you should in getting it. The LCD is listed at $7.60 at the time of this writing. Shipping on that item was over $10 for me – way too steep for such a part. It actually made more sense for me to purchase an entire second DSO 138 kit in case I screwed something else up too (which I didn’t, thankfully. ;-)).

After I finished assembling everything, I did a quick test with the built in 1KHz 3.3 V test signal. That checked out ok. I then did a lengthier test using my favorite signal generator – my Sansa MP3 player. I fired up a Foo Fighters tune and watched the display dance around for a while.


I’ve still got some work to do in learning the DSO 138. When purchasing this kit, I also picked up a cheap high frequency signal generator kit. At the time of this writing, I haven’t assembled it yet. But once I do, I’ll probably write a short article about it as well.

What do I think about the DSO 138 thus far? It’s definitely worth the $25. Assembly was fun and gratifying. For low frequency signal sources, it seems to be fairly accurate. And the electrical noise level is surprisingly low. I definitely recommend it to everybody. Even electronics gurus will find one of these handy to keep in the toolbox (it’s portable!).

But as LeVar Burton says, “You don’t have to take my word for it”. Check out the DSO 138 review done by GreatScott!

GDB Tips and Tricks #3: Saving and Restoring Breakpoints Using Files

You spent the last 10 minutes littering your application with breakpoints in all the places where you think bad things might be happening. Then you run your application. Maybe you missed something. Maybe you didn’t. But now after a few minutes of debugging, both your application and GDB appear to be in a funky state. What you want to do is just quit out of everything and start fresh with a new session. But what about all those breakpoints? Typing in the “b” commands was such a chore. Even if you could remember where they all were, the mere thought of doing so is exhausting. And what if you need to start over yet again? Surely there’s a better way.

Did you know you could save your breakpoints to a file? All you need to do is issue the “save breakpoints” command like so…

(gdb) save breakpoints my.brk
Saved to file 'my.brk'.

This will save all types of breakpoints (regular breakpoints, watchpoints, and catchpoints) to the file my.brk. You can in fact name the file whatever you’d like.

Later when you’re ready to reload your breakpoints, you can issue the source command.

(gdb) source my.brk

There’s nothing special about this breakpoints file. It’s just a text file containing a list of gdb commands separated by newlines. In fact, not only can you amend it manually with more breakpoint commands, but you can add in just about any other gdb command in the process.

It’s worth noting that the “source” command isn’t actually specific to breakpoints. It will execute anything in the specified file as written, which makes the “source” command a handy tool in its own right.