Tired of writing shell scripts with 30 years old technology? So were we. Welcome to NYoSh: Not Your ordinary Shell.

NYoSh is designed as a modern replacement for Unix/Linux command line shells (BASH, TSCH, C shell, KSH, Bourne shell). NYoSh is built as a collection of languages implemented in the open-source Jetbrains MPS language workbench.

Fall 2014: NYoSh is now part of the NYoSh Analysis Workbench, an interactive data analysis platform built with the MPS language workbench.

Rationale: Our lab, like many others, uses BASH a lot to write scripts that execute programs and automate analyses of biological data. Recently, we developed GobyWeb where BASH scripts are used to implement analysis plugins. GobyWeb has a lot of plugins. This means we wrote a lot of BASH scripts. BASH scripts get the job done, but guess what? It is not easy to write robust BASH scripts. More importantly, we came to realize that we were spending a lot of time writing BASH scripts and maintaining them, because the BASH language  is lacking many features that you have come to expect in modern languages. We are also working on analysis pipelines for clinical applications, where error detection and reporting is critical. BASH is not a good language to work with if you need to detect error conditions reliably. If you are writing BASH scripts, you are using a language that was designed in the 80s. Yes, that is 30 years ago, and a lot has happened in computer science in these 30 years!

We are not the only ones who thought BASH scripts are a necessary evil. Others have noticed and discussed alternatives. None of these alternatives are really good, in our opinion, because while they provide extensibility, they

(1) make it harder to do things that were simple to do in BASH (e.g., running command lines with pipes and redirection should be easy in a useful BASH replacement language, or matching filenames)

(2) lack an integrated development environment (IDE).

NYoSh is our answer to these challenges. NYoSh is implemented as a collection of languages that are designed to work together in a IDE (see Design section below). These languages support:

  • A command execution language. This language makes it easy to  execute commands in a similar fashion to standard shells (e.g., running commands separated with |, &&, ;, |& or ||). You can get the standard output of a command pipeline as a stream or reader and process it further.
  • A groovy-like string language, where you can embed references to variables: “literal${a}2” will evaluate to the string “literalA2” if the variable string a was defined with the value “A”.
  • a filename matching language. This language makes it possible to expand  wildcards into filenames. For instance “*.xml” will expand to all filenames matching *.xml in some directory. *.xml – *1.xml will identify all filenames ending in .xml except those filenames ending in 1.xml.
  • An environment language. This supports loading environments variables in a BASH compatible format (VAR=VALUE format). You can export variables to subprocess started with the command execution language.
  • A steps logging language. This logs a record of execution steps, with success or failure of each step. The steps language helps script developers document the steps taken by the script, and which commands are executed within which steps.
  • Base language. This language is provided by MPS and on the surface is similar to Java. Base language provides all the usual control statements such as for loops, if statements, etc.

In contrast to Java, the MPS base language can be extended with new types of statements and expressions. We use this feature to combine the above languages into NYoSh, a modern shell scripting language.

Innovation. The following innovations make NYoSh really different, and make you fundamentally more productive:

  1.  The NYoSh Workbench is a full featured IDE built with Jetbrains MPS and the NYoSh language. You can write NYoSh scripts with a full-featured modern IDE. Benefits include undo/redo support, context sensitive code completion, seamless recompilation and execution, refactoring, intentions, tight integration with source control systems (Git, Subversion, CVS)
  2. NYoSh scripts compile to byte code that runs on the Java Virtual Machine 1.6+. You can execute the scripts on any platform that supports Java 1.6+ (portability of specific scripts is only limited by the kind of programs used in command processes).

Design.We have designed NYoSh in the Meta Programming System. MPS is an open-source project of JetBrains, the company behind Intellij IDEA, the market leading Java IDE. MPS is a language workbench that makes it possible for language designers to create their own language. Once designed and implemented, these languages can be used directly into IDEA, or packaged in a standalone IDE. We have taken the later approach with NYoSh. See the following pre-print for more details:

Publication.  

Simi M, Campagne F. (2014) Composable languages for bioinformatics: the NYoSh experiment. PeerJ 2:e241 http://dx.doi.org/10.7717/peerj.241

Forum. Please direct questions about NYoSh to the user forum.