One Small Program's Journey To 1.0.0
This is the story of Qodem.
Qodem's story is typical of many small open-source applications. It
was originally started to "scratch an itch", and then got better
slowly until it finally reached its lofty goal of a 1.0.0 release. It
has the distinction of being both the most complex and the
longest-running project I have ever worked on, with some of the
original code surviving almost fully intact through it's entire
development lifespan of fourteen years. This document is my
post-mortem of Qodem's 1.0.0 release: some things I'm proud of, some
not, some things I rightfully broke convention on, and some things I
had to stubbornly learn the hard way. I write this to help remember
for myself, and also for its therapeutic value.
What I Wish I Had Done
~~~~~~~~~~~~~~~~~~~~~~
1. GNU Hello World
Let me start with two of the most useless time sinks: autoconf and
gettext. Had I just taken the GNU Hello World example and begun
growing from there, I would have had both autoconf/automake and
gettext from the get-go. Instead, I started as a "simple C"
program with my own Makefile and switched later to
autoconf/automake, and I also started with my own plain US-English
strings and later switched them to ALLCAPS keys (similar to how we
had been using Java ResourceBundles at work), and then later
switched the translation keys back to en_US strings. This was
three steps that each involved a code sweep through 10-20 KLOC
just to get me back to what I had been used to as a user before I
touched anything.
The good things about autoconf/automake/gettext are:
* Both the Debian-based and RPM-based distributions know how to
handle it.
* It is much nicer to use --enable-X than hand-editing a Makefile
or passing -DSOME_FEATURE to make.
* Cross-compiling with mingw32 is very easy.
* 'make dist' . 'Nuff said.
Moral: Every new open-source project should start with GNU Hello
World.
2. Abstracted my use of ncurses at the beginning
When I first started, I wanted nothing to do with Windows or X11.
This was meant for the raw console with standard VGA colors and it
would be a Linux/Unix project forever. So I didn't mind liberally
sprinking direct ncurses functions and ncurses values (KEY_X)
throughout the project. I also made use of the ncurses forms
library directly. Oh but then....
Then I wanted to switch to the wide-char ncurses, and I discovered
that the include file is in different locations for narrow and
wide-char ncurses depending on OS, and that I could not mix
wide-char ncurses calls with narrow-char calls, and the forms
library broke on OS X. And that was all before I added support
for PDCurses, which exposed many other bad things I was doing. I
even had to stop using 'bool' before I was finished! In the end
starting out with curses calls directly in my code led to some
VERY big code sweeps, replacing the forms library with my own
simpler code, and even using my own include header just to include
curses.h the right way! But despite all this work, I think
essentially all of the curses bugs were mine, not theirs.
My current code is nice and clean. curses calls are tucked away
into one user input module and one screen output module and it's
wchar_t everywhere in the calling code. Porting to Win32 was
ultimately a much smaller effort because of this.
Moral: Abstract away all library use from the very beginning.
3. Re-used other code
If I had started this project just two years later, I would have
been able to use Synchronet's sexyz when it was first released in
June 2005 instead of writing my own Zmodem in ... January - June
2005! Similarly I found large chunks of function floating out
there long after I had wrapped my own version. Other examples are
libvterm and rhtvision (which is sadly not considered free
software).
But besides the fact that time travel is impossible, I also claim
in my defense that most of the other code I could have used didn't
fit into the overall structure of Qodem. Qodem is more than just
a capable VT100 emulator, it's also ANSI.SYS, Avatar, DEBUG, and
then the other functions like file transfers and scripting.
Some Other Morals
~~~~~~~~~~~~~~~~~
1. Never blame your library or another application (except Zmodem).
I cannot recall a true bug in ncurses through this entire project,
with the possible exception of how it handles a mix of narrow-char
vs wide-char (but that might be me too). In general, ncurses has
behaved exactly as advertised, and it's been me doing something
wrong when using it.
Similarly for xterm: despite many many horrible-looking screens I
had to fix, I really only encountered one bug while using xterm.
An invalid windowOps sequence caused my X server to crash. That
one might have been xterm's or the X server, I'll likely never
know, but it had been fixed the next year when I looked for it
again.
But MY bugs, well there were many to choose from.
Zmodem is my exception to this moral. The protocol itself is
broken in several ways, and the reference implementation in
particular (rzsz) is very bad in behavior and generally unreadable
for learning. If anyone out there implements Zmodem, expect to do
a LOT of testing with other implementations. You might do very
well by either taking sexyz or modifying mine to suit you.
2. Stay caught up with modern Linux.
I started this project in 2003 on the text console of Debian
Woody. I spent over a year at the text console of Slackware 9.x.
In those days the console was 8-bit CP437 by default with true VGA
colors (i.e. yellow is brown, bold yellow is yellow). Lots has
changed since then.
I spent basically two and half releases getting Qodem running on
modern systems: wide char input and output, background color,
screen dimensions, VGA colors, Unicode (which isn't the same as
wide char), and fonts. I also wrote a Getting Started Guide
outlining all of the isues. Turns out, essentially all of those
issues are solved by better understanding of curses (pun intended)
and the user doesn't have to see anything.
3. Implement everything that nags you.
Procomm for Windows has a neat feature called the "monitor window"
in which bytes sent across the wire are displayed in a hex dump
format very similar to Qodem's DEBUG emulation. (And I was so
proud to do DEBUG in a different way than Qmodem(tm).) In the
monitor window, bytes sent to the remote system are colored
differently from bytes received. I knew about this feature for
about four years and never bothered implementing it, and then one
day I sat down and did and it was about 15 lines in total to do.
And now it's in, it looks good, and I never have to think about it
again.
Open source is driven differently than any other kind of software
development model. They call it "stone soup": you put in one
thing at a time and after a long time it's really good. There's
no time or money pressure to prioritize certain features over
others. There's no need to let something drop because you'll
never have time to get to it.
When you see something you'd like, just put it in sometime.
You'll feel better.
The Story Of Qodem
~~~~~~~~~~~~~~~~~~
Now that the lessons are over, let's get to Qodem itself.
The most useful parts of Qodem weren't in until version 0.0.6. Prior
to that it was most definitely a tiny toy, showing some Qmodem theme
colors and screens but really just barely wrapping a shell. Yet it
was still ~30 KLOC -- pretty big for not too much. OTOH this was
enough to visit BBS's, pass most of vttest, and even do Zmodem via
rzsz. This was the point at which I had a comfortable grasp of the
Linux programming API: files, ioctls, ncurses, etc., and could start
to basically "make anything happen given enough time". Most projects
could stop around here and be a nice little gem.
But of course I didn't stop there. I had a roadmap to get from where
I was to full feature parity with Qmodem, and that meant pushing
forward off and on. The next year and half and 15 KLOC was mostly bug
fixes and polish on what was there already but also keyboard macros,
internal transfer protocols, and breaking out of the forkpty()-only
world and adding real serial port support. 0.1.2 was another nice
spot to stop at, and I mostly did stop for a year as my personal life
took priority.
The next 10 KLOC were again polish and bug fixes, mostly around
Unicode handling. At 52 KLOC I had broken out of the Linux console
and looked good on Unicode terminals. 0.1.3 was a very nice release,
and probably the best "sweet spot" in Qodem's history. About 90% of
all of the technical fun of this project was finished by this point.
From here on out it was a morass of polish and bug fixes, as I
essentially switched hats from developer/engineer to QA/engineer and
ultimately project manager.
Only 2 KLOC later at 0.2.0 I was into my first packaging nightmare:
Debian. Qodem benefitted enormously from it in the end, and this
experience actually gave me a hook for about six months of work in a
very nice city several years later (packaging a commercial SaaS
product for internal deployments, which also taught me RPM which is
MUCH BETTER on the build and test side), but it was my first real
encounter with the outside world. Honestly, it mostly sucked. I put
Qodem on Freshmeat (remember that?). I created a Wikipedia page and
almost immediately came to regret it (it wasn't notable you see), an
experience I think SyncTERM ran into also. I tried hard to get Debian
to get Qodem in the repository but they never bit (despite me doing
all the actual packaging work and passing lintian) even though it is
in Fedora. I started to make the mistake of taking Qodem more
seriously and linking it back to my technical self esteem. This is
the spot where someone a little more sane than I would have just moved
on to something else.
But I really liked my old DOS-based Qmodem, didn't want to see its UI
die yet, and was depressed at my day job thus had a bit of motivation
to do SOMETHING right. So I started with small things, kind of
scratching my itch again, and the one really big thing that had always
stymied me (scripts) suddenly had an easy solution. 0.3.2 came about
8 months after 0.2.0, only another 2 KLOC (58 KLOC in total), but it
was in most ways my favorite single release. It knew Linux, serial
ports, forkpty(), and ncurses, and that was it. It had no other
network protocols, operating systems, build systems, installers,
... it was right on the edge of the Uncanny Valley. It was also right
on the line of what my puny brain could hang onto at any one time.
From this point on significant refactorings would be much harder due
to the many ways Qodem's features could step on each other.
The next release was at the bottom of the Uncanny Valley. Windows
"worked", but only via cross-compiler. Network protocols for telnet,
rlogin, and a ssh library were present, but they weren't quite as
convenient as the forkpty()'d clients. An X11 build was present and
looked amazing, but it was buggy as hell. The codebase at 80 KLOC was
37% bigger, yet the release felt shakier and less fun than before. We
were entering bad territory. Worse, I had taken almost a year and a
half to get there: my new job was a lot more fun and I didn't have a
bunch of depression and anger fueling this hobby anymore. Qodem is a
teeny tiny thing compared to Netscape, but it went through a very
similar pain: 1.0alpha felt much like those early Mozilla 0.9x builds
that felt so much worse than Netscape Communicator 4 despite being
newer and theoretically better. But unlike the Mozilla Foundation, I
didn't keep fighting my way through after the 1.0alpha release. I was
exhausted from pushing over 20 KLOC in short spurts and making Qodem
too big to fit in my one head anymore. When I left the steaming pile
of crap that is 1.0alpha on SourceForge I took what I thought would be
just a couple months off, but those months turned into THREE YEARS....
. . .
I honestly thought I was finished with Qodem at this point. No one
seemed interested in it, no bug reports had come in, the few
references to it on DOVE-Net were basically "no one has seen the dev
in forever, use SyncTERM instead", and SourceForge had trashed its
reputation with bullshit malware installers. I had scavenged some
good stuff from it for other projects (Kermit, codepages, VT100), and
was ready to just leave it behind. But a couple things aligned to get
me interested again. I had started a new project to build a compiler,
and my first target was DOS, so I was playing with Borland compilers
again; I was used to GitHub and it was much nicer and easier than
SourceForge; I had done some RPM builds and could clean up Qodem's
spec files; and the big one was that I resurrected my old Win2k VM to
solve some other problems and noticed Borland C++ 5.02 just waiting
for me there.
So ... I wondered how hard would it be to lift Qodem out of its crufty
SourceForge repository and start fresh with a clean build. I slowly
got each source file to compile on Borland, fixed many compiler
warnings, fixed many actual crashes on startup, and then saw it
running as a true Win32 application. My wife had an evening job which
gave me plenty of time to slog through dozens of small things. I
performed my last major code sweep, recognized that I no longer had
the time to ever do a project of this scale again, and decided to put
everything in the public domain. I was going again, but this time
quietly.
In a happy incident the world noticed me working on Qodem before I had
really announced anything. It was a really nice moment, someone out
still had some interest and seemed excited that I was going again.
This time I paced myself. I knew at the beginning of this cycle that
this was going to be far less about new function, almost no real "dev
fun". Instead this was going to be a slow-paced packaging and
polishing exercise. The hardest part by far was cleaning up the
dependencies for the Windows build and debugging the subtle
differences between the various ncurses/PDCurses libraries. But very
little in the way of crash bugs, screen corruption, file transfer
errors, etc. After Windows looked pretty good and I had the InnoSetup
installer running, I told the world that 1.0beta was coming soon and
please bang on it. This time a few people obliged, and I got their
issues fixed.
The new code style sweep slightly increased the line count, but
refactoring brought it mostly back in line with what it used to be.
1.0beta was clocking in at 94 KLOC. It had four Linux package builds,
two generic Unix builds, three Windows builds, a Windows installer,
and a Mac App Bundle. It officially supported two native Windows
compilers plus gcc. It even has a real icon.
Qodem was just about ready to go into maintenance mode. 1.0beta was
released one week shy of its thirteenth birthday. It was a very
decent release, so good that I was almost ready to use it as my main X
terminal. With this new baseline I discovered a couple dozen small
paper cuts to fix and proceeded to work on them. I had users on OSX,
Windows, and BSD submit bug reports, and even had some pull requests
come through!
Life intervened as it likes to, but now I truly am in the final
stretch. I'm not doing it anymore for pride, or to escape my job, or
burn time, or even scratch any more itches. I'm doing it to FINISH
IT. I just want the personal satisfaction of knowing that I climbed
out of the Uncanny Valley and have laid this one to rest in a good
state. Qodem 1.0.0 has grown only a little in code size (5 KLOC, now
a grand total of 99 KLOC), but gained another round of paper cut fixes
and is now good enough to be my main terminal.
Perhaps I will continue adding features to Qodem. I don't have to, it
is pretty awesome now. But there are some items on the wishlist that
would be nice to have someday. In the meantime I will try to be a
good upstream and fix whatever bugs are reported.
What Have I Become?
~~~~~~~~~~~~~~~~~~~
Somewhere along the way in doing this project I seem to have become a
bit of an expert in text-mode GUI systems. I know vttest, ANSI music,
and X10 mouse reporting. I know Zmodem and Kermit, (parts of)
Unicode, and PDcurses. I've liberated bits of Qodem: created a Turbo
Vision clone that speaks both Swing and Xterm, and started a brand-new
Java implementation of Xmodem/Ymodem (and later will be Kermit and
Zmodem) that also allowed me to finish up the /G protocols in Qodem
finally (since rzsz doesn't honor /G and I couldn't really test them
before).
I have become an architect too. I know when things "feel" about
right. As I re-work a number of these bits in new Java projects I
find myself choosing dramatically simpler implementations -- of course
the huge Java standard library helps a great deal -- but also because
I am willing to cut the critical 5% that reduces the workload by 80%.
And I know how to say no: some of the pull requests were taken as-is,
but most were largely re-worked in order to fit in the rest of the
super-structure. (Because Qodem is an unusual thing. It's not an X11
terminal, or a protocol, or a VT100. It is the union of all of those,
but also more. In truth it is far closer to tmux than it is to
minicom.)
I feel like I have done more here than just crank out over 100 KLOC
for a small program. I have fond memories of visiting the NC State
University library and reading up on "teleprocessing terminals" from a
book circa 1985, and the excitement of making TTY be more than just a
"dumb" terminal. Two weeks later we had just gorgeous weather in
Raleigh, and I would code the keyboard macro editor screens for a
couple hours and then take our kitten outside to eat grass and get
dirty. Among my many used book hauls were copies of "Yakety Mac",
"The Complete Handbook Of Personal Computer Communications", and many
others. I have watched the BBS Documentary God only knows how many
times. This wasn't all just code, it was total immersion in a world
that I cannot help but feel sad is passing away.
The Final Bit
~~~~~~~~~~~~~
All of the small programs we use have traveled this similar path.
There was a time when rsync, or tmux, or you-name-it, was just awful,
and then years later became OK, and then much later became its own
special thing. Some of these will enter the regular lexicon, perhaps
even being included in the default install of a distro. Meanwhile
others will remain obscure masterpieces, uncertainly awaiting
discovery by a future generation of digital archaeologists.
For me, this marks the end of a very long journey. As a teenager I
always wondered how in world Qmodem could do what it did. Take
Zmodem: I just couldn't wrap my head around communicating with the
modem, updating the screen, writing to disk, handling CRC, and all the
other details. Ditto for the ANSI processing. Yet now I know exactly
how one might do it. (I even have the many hours of dreadful memories
digging through debug logs from both qodem and rzsz trying to figure
out just what in the hell was going on. I still feel some fear when I
am forced to dig into the protocols.)
It is impossible for me to not feel emotional around this program.
Qmodem was my window to the world, what connected me in an important
time in my life to the only people to whom I could relate. Even
though the world has long since moved on from both dialup and
text-based terminals, I remain happy that Qodem can live on, for now,
in modern systems. Somewhere in all this mush I also remember our
beloved and departed cat Hazel. Some of this urge to finish I think
stems from missing her.
After 1.0.0 is released, bugs and all, I intend to typeset it and put
it on my bookshelf. It will be my personal reminder of the memories
of doing this. At one time in my life, I really did put in the
effort, and I made something interesting.