Lessons Learned Building JFrog Platform Installers: Applying the Template pattern to Shell Scripts
Artifactory, JFrog’s flagship product, has been in the market for more than a decade now. Individual developers working on Open source projects, Startups - big and small, and large Enterprises all use it to store binaries. Over time, Artifactory has shipped with every installation type imaginable: from ZIP files, to native installers & Helm charts.
Over the years - to further our vision of Liquid software, we’ve launched other products: Xray, Mission Control & Distribution in this ecosystem.
As each product matured, their installers began to rely on interactive installation scripts.
Interactive installation scripts offer an easier experience to an end user. To product teams, they are attractive because they:
- Offer finer grained control over the installation experience
- Reduce need for elaborate documentation covering all possibilities
- Make user errors less likely
Last year, at SwampUp 2019, we announced the new JFrog Unified Platform. Our aim was to provide a unified experience to all users of our products. One of these experiences was the installation of our products.
From an installation perspective, our goals for the installation experience were that each be:
- Backward compatible
- Easy to use
- Unified (Similar, work well with each other)
With these in mind, all our products now ship with interactive installation scripts. The one notable exception is the native installation package for Artifactory - which, we felt was simple to install.
The experience of building installers for JFrog products was challenging and enriching for my team and me. We learned much during the process about customer focus, enhancing support-ability, design and user experience.
The rest of this blog deals with one specific set of lessons: those learned from applying the Template pattern to shell scripts.
Template Pattern
At a very high level, what an interactive installation script does is very simple and is demonstrated in the diagram below.
Things get complicated when we get into details.
- Each product has its own microservices. (Read more about our System Architecture here)
- Each uses different databases. For instance, Xray uses RabbitMQ for messaging, Distribution uses Redis and Mission Control: ElasticSearch. (Read more about the Databases we use and support in Common Resources)
The problem statement then is
“We have a common sequence of operations but each step in the operation is implemented differently“
This, we realised, was simply the Template pattern.
Template method is one of the behavioral design patterns identified by Gamma et al in the book Design Patterns.
The template method is a method in a superclass, usually an abstract superclass, and defines the skeleton of an operation in terms of a number of high-level steps. These steps are themselves implemented by additional helper methods in the same class as the template method.
The intent of the template method is to define the overall structure of the operation, while allowing subclasses to refine, or redefine, certain steps
Most programming languages, such as Java, have support for class hierarchies and abstract methods which make implementing the template pattern straightforward. Bash, the language we wrote our interactive scripts in, does not have the same language constructs. Therefore we had to improvise. However, it turned out to be fairly simple.
Template Pattern - Approach 1
One approach was to simply encapsulate the sequence within a generic script with a main method. The user would invoke the generic script. The script in turn, would invoke the product-specific hooks.
Notice that in this approach
- Each hook needs to be explicitly coded for. The product-specific script cannot decide what methods it wants to override.
- Each hook has to be implemented (even if with a dummy implementation) in the product-specific script. Not doing so will result in a runtime error.
- Variables available to the parent script, unless they are also exported as environment variables, are NOT available to the product-specific script and vice-versa.
This approach is particularly suited (and even recommended) for scenarios where you are invoking a third-party script - one whose internals you don’t care about and with which you don’t want to share too much of your internal state, etc.
Template Pattern - Approach 2
Consider an alternative approach, depicted by the sequence diagram below:
- Here, the user invokes the product-specific script. This allows us to bootstrap with product-specific variables before beginning the common sequence of operations
- The product-specific script
sourcesthe generic script and invokes its main method. This allows variables to be shared across both scripts. - Sourcing also facilitates
method overriding. Methods in the product-specific script, which appear below the sourced script’s methods, override the ones in the generic script. - This approach also avoids unnecessary code duplication. The generic script already contains dummy or actual method implementations. Only hooks which need to be implemented in the product-specific script need to be added to it.
This approach is more suitable when you own both scripts and want them to share information.
Conclusion
Implementing the template pattern allowed us to control the overall installation experience while being able to accommodate every product’s particular quirks. It made our design more agile and even allowed us to easily incorporate changes later in the development cycle, from our beta users.
Do let me know in the comments if you found this blog’s contents useful and would like to know more about the other lessons we learned.
References
- Template Pattern
- The Flow chart was created using https://www.draw.io/
- The Sequence diagrams were created using https://www.websequencediagrams.com/