MiniCalc, a Delphi-like scripting language, is used heavily at First Trust. It dates back to late Feb. 2001, with humble beginnings – it wasn't until 3 years in (~100 revisions) that we decided to expand it beyond simple expressions (supporting loops, conditionals, functions, exceptions, etc).

The primary use cases for MiniCalc are…

• DSConfig files
  (~900)
• DB Configs
  (~200)
• DB Scripts
  (~300)
• VDB Explorer
  • exec and vdb statements
  • custom action buttons
    (~20)
  • startup scripts
    (~25)
  • profile factories
    (~4)
• Syntax Editor
  • templates
  • scripts
  • paste loop command
  • Macro | Repeated Playback command
  • custom run commands
  • active documents
• Shadowfax deployment scripts
  (~400)
• Rinse
  • calculated fields
    (~35)
  • conditional directives: {$IfDef}, etc
    (~8,300, but lots of false positives in {$begin_inline} blocks; also, the vast majority are in code generated by GenevaXsdToRinse)
  • dynamic code generator: {$Calc}, {$begin_inline_calc}, {$end_inline_calc}, etc (not fully documented yet)
• VDB: Macros
  (~30)
• Filters and comparisons for prop bags (superseded by TVarExpr)
• CodeVault
  • Project Templates
    (~9)
  • Links from emails, etc (code review requests, etc)
  • File Wizards (Revision | New | ...)
    (~4)
• Hermes
  • schedule expressions (triggered events)
    (~250)
  • schedule expressions (scheduled events)
    (~200)
  • action scripts
    (~620: 288 Group + 40 Src + 245 Dst + 50 SQL)
  • SQL events
    (~165: 120 Read + 45 Update)
  • dynamic command-lines
    (~900)
  • dynamic file names (delimited with pipe characters)
    (~70: 25 SrcPath + 10 SrcFile + 36 DstFile)
  • agent downtime scheduling
• World Date Service: custom calendars
  (~110)
• Smart Dialog: the NullExpr property
  (~160)
• Sage
  • Dynamic Content with MiniCalc
    (~190: 81 Early + 110 Late)
  • Fluent API for diagrams
    (~30)
• DSPreprocessor (used by Bifrost to support TypeScript)
• Orion compound filters
  (~25)
• DSRttiCalc
  (~220)
• Bolt selection scripts

The above list of three dozen or so is not exhaustive, but captures the most important use cases.

Some observations based on the above information…

• Few of the above could be accomplished without a scripting language of some sort.
• Using the same language for each of the above scenarios is certainly helpful.
• That the language is Delphi-like is probably helpful.
• That we control the language, and that it nicely integrates with Delphi in general, and our architecture in particular, is important for nearly all use cases.

Although MiniCalc has been extremely effective and flexible over the years, it does have a number of serious drawbacks…

• Dynamic Typing
  MiniCalc catches at runtime issues that Delphi would find at compile time, including…
  • Misspelled identifiers (variables, functions, methods, etc).
  • Wrong number of arguments passed to a function.
  • Wrong types of arguments passed to a function.
  • Type mismatches with operators.
  Additionally, MiniCalc suffers from several questionable design choices from 20+ years ago…
  • Reading an undefined identifier yields unassigned, which quietly converts to zero / false / empty string / etc.
  • Writing an undefined identifier creates a variable (does not apply within functions).
  • MiniCalc includes the comma operator inspired by C, which has proven to be a frequent source of errors.
• Differences with Delphi
  We treat MiniCalc as being Delphi-like, especially when running Fluent SQL in VDB Explorer. However, MiniCalc has a worrying number of gratuitous incompatibilities with Delphi, including…
  • Different operator meanings (notably @ and &).
  • Different operator precedence (notably logical and relational).
  • The whole @-sign nightmare.
  • MiniCalc uses dynamic scoping, while Delphi uses static.
  • MiniCalc defines several keywords/operators that are not keywords in Delphi: bitand, bitnot, bitor, bitxor, defer, false, inceptive, null, true, unassigned, and using.
  • MiniCalc does not support generic types: you can call @NewList or @NewIntegerList, but you cannot instantiate an IList<Boolean> or an IList<IStringList>.
  • Enums are strings in MiniCalc, making comparisons dangerous.
  • Delphi sets are lists in MiniCalc.
  • Many Delphi records become custom Variants in MiniCalc.
  • MiniCalc requires empty parentheses to call zero-argument functions and procedures.
  • MiniCalc functions do not have the familiar Result variable.
  • In MiniCalc, everything is an expression, even statements like while, for, and try / except.
• Maintenance Issues
  MiniCalc has a couple of maintenance problems…
  • Exposing methods and routines from Delphi into MiniCalc is tedious, messy, and error-prone.
  • Documenting the types, constants, routines, and methods available to MiniCalc code is a tedious manual process, and the end result is never quite complete or accurate.
  • A good portion of the complexity of MiniCalc's implementation is devoted to implementing all those differences between Delphi and MiniCalc, for example 34 additional operators not present in Delphi!

Some of these shortcomings are quite serious (operator precedence, @-sign nightmare, required parentheses for function calls), while others are minor enough in isolation (absence of Result).

Taken together, these issues lead us to conclude that a next-gen version of MiniCalc is long overdue. If this can happen before Jay & David retire, it will put the company in much better shape than if MiniCalc remains as is.

The above shortcomings of MiniCalc are addressed by Tusk, a next-generation iteration of MiniCalc. This section outlines the high-level benefits of migrating our MiniCalc code to Tusk.

• Tusk is statically typed, making it much safer than MiniCalc…
  • Easier to get a script up and running initially
  • Easier to test scripts as things evolve
  • We envision nightly tests that parse & analyze all DB Scripts, all .tkconfig files, all Hermes action scripts, all Hermes SQL events, etc.
  • The validation performed when checking .tusk or .tkconfig files into CodeVault will be much more valuable
• Tusk is much more similar to Delphi than MiniCalc…
  • Lessens the learning curve for new developers
  • Reduces the mental cost of knowing / using the scripting language
  • Facilitates sharing code between Delphi and Tusk, especially Fluent SQL in VDB Explorer
  • Simplifies the documentation
  • Makes the scripting language less error-prone
  • Eliminates redundancies / complexities such as defining a custom Variant type for TSQLExpr (and related records) because MiniCalc (unlike Tusk) doesn't support records
• Documenting runtime libraries (functions, methods, constants, etc) is a manual process for MiniCalc, but machine-generated for Tusk
  • Much less work
  • Much more complete & accurate
• Exposing global routines and interface/record methods is a complicated, error-prone process for MiniCalc, but is much simpler for Tusk (example below)
• In many ways, Tusk improves upon Delphi; highlights include…
  • DateTime Conversions
  • Improved Operator Clarity
  • Trailing Commas
  • Optional Declarations
  • Indexed for…in Loops
  • Strict case Statements
  • DSFormat Validation
  • Data Types
  • Late Evaluation
  • Function Results
  • Properties
  • Any and NoCall Types
• If Tusk replaces MiniCalc before Jay & David retire, that gives the Enterprise team a much easier project to evolve and maintain. MiniCalc was a solo effort (Jay only), while Tusk is a co-op project (initially Jay and Tom; joined later by Mitchell and Giorgi).

For more details on some of the above issues, see the following…

• Overview of the static analysis that Tusk performs.
• Specific ways in which Tusk offers better Delphi compatibility than MiniCalc.
• An example showing how exposing routines is simpler and safer in Tusk than in MiniCalc.
• Key ways in which Tusk improves upon Delphi.

Below is an example illustrating the differences between MiniCalc and Tusk, when exposing a Delphi routine/method to the scripting language.

This is how Pos is exposed in MiniCalc…

// MiniCalc... class function MCExpose.mcPos(var Info: TMCCallInfo): Variant; var V1, V2: Variant; t1, t2, Offset: Integer; begin Info.CheckCount(2, 3); Info.NextArg(V1); Info.NextArg(V2); Offset := Info.NextArgDef(1); t1 := VarType(V1); t2 := VarType(V2); if t1 = t2 then begin case t1 of varString: Exit(Pos(_VarGetAStr(V1), _VarGetAStr(V2), Offset)); varUString: Exit(Pos(_VarGetUStr(V1), _VarGetUStr(V2), Offset)); varOleStr: Exit(Pos(_VarGetWStrPtr(V1)^, _VarGetWStrPtr(V2)^, Offset)); end; end; Result := Pos(string(V1), string(V2), Offset); end;

This is how Pos is exposed in Tusk…

// Tusk... class function Expose.tfPos( const SubStr, Str: string; Offset: Integer): Integer; begin Result := Pos(SubStr, Str, Offset); end; class function Expose.tfPos( const SubStr, Str: AnsiString; Offset: Integer): Integer; begin Result := Pos(SubStr, Str, Offset); end;

Note how the MiniCalc version must determine which overload to call, based on the number and types of arguments. Also, note how the MiniCalc version allows things that aren't allowed in Delphi, such as…

@Pos(123, 991239)

The above returns 3. Tusk would flag this as a type mismatch (early).

Eliminating this kind of code is one of the motivations for moving to Tusk.