However, these approaches are, at best a minor enhancement to a more general solution. We will never be able to have all our documentation come from source code. These automatic approaches are well suited for APIs and utility routines. Specifically, they are good for discussing the general purpose of a type or method, and for describing the arguments and results of functions.

However, these tools are a poor fit for many other kinds of documentation, such as…

• High-level programming concepts, such as threads and fibers, lexers, parsers, hash tables, and so on.
• Tutorials, where we lead the reader through a sequence of examples that build one on the other.
• Troubleshooting, pitfalls, best practices, etc.

Let's take some concrete examples that may be familiar to Delphi programmers at First Trust.

• Should the unit DSMiniCalc.pas be the home of the MiniCalc documentation, discussing the language syntax, library routines, and so forth?
• Should the unit VirtualDB.pas contain the entire VDB Tutorial (including Fluent SQL)?
• Should DSThreadUtil.pas (and related units, such as DSFibers.pas and DSLock.pas) hold the Threads and Fibers documentation?

Of course, there are some appropriate uses for this technique: routines in DSStringUtil, for example. Functions like DSFormat, EncodeIdent, and ParseCSV seem like obvious candidates. However, there are even things in this same unit that need separate (and voluminous) documentation. For example, the type TDSCharSet occupies about 150 lines, for the type definition (three to five screens full of source code). Yet, the documentation for this type is 15 printed pages, and has a top-down, logical structure, which bears little resemblance to the type definition.

This brings us to another thorny issue: how to merge the two streams of content. Suppose we have a bunch of information that is gleaned from source code, and even more residing elsewhere. How do we weave it all into an organized and coherent collection of documents? For example, how would we document the VDB library if some of the details come from source code, but the lion's share is in external documents? It's much simpler when the documentation is all in one place: you can see (and change) how it's organized much easier.

None of this is to say that a tool that automatically creates documentation from source code can't work, or isn't worth the trouble. But, we do feel that this idea should wait until we have a good amount of mileage with Sage. Perhaps down the road, we will figure out a good approach – one that strikes the right balance between the competing concerns of (1) having thorough, detailed, and complete documentation, and source code that isn't peppered with huge comment blocks that obscure the actual code, and (2) keeping the documentation adjacent to the code it documents.

However, we will proceed cautiously here, in part because we have already tried this approach in the past. For several years, we used the tool Doc-O-Matic, to produce documentation based (in part) on parsing source code (and adjacent comments). We found that, while it essentially did work, it wasn't at all pleasant. We ended up with very messy interface sections in our Delphi units, and in exchange, we got documentation that was difficult to organize and restructure. It works, but it doesn't work well.

In contrast, with Sage, we can write whatever documentation we think the topic at hand requires, without cluttering up the source code. Yes, we will have more duplication: the comment above DSFormat will have information that's duplicated in the corresponding .sage article. However, that article will be more thorough, with more examples and details, than we would ever want to dump into a source code comment. This is the lesson we learned (the hard way) from using Doc-O-Matic.

Even though we don't have a solution like Doc-O-Matic, Sage offers some Delphi and DSEdit integration features that narrow the gap.

So, while we don't (yet) have a way to keep Sage content in comments, we can at least have convenient and conspicuous links to the content.