# Introduction

DCli is the console SDK for Dart.

Use the DCli console SDK to build cross platform, command line (CLI) applications and scripts using the Dart programming language.

The DCli (pronounced d-kleye) console SDK includes command line tools and an extensive API for building CLI apps.

The DCli console SDK as featured on Jermaine Oppong package of the week vlog.

{% embed url="<https://youtu.be/z99IxxWmD1Q>" %}

## Sponsored by OnePub

Help support DCli by supporting [OnePub](https://onepub.dev/drive/a50d4f6f-e0fb-40bd-af7b-2dcc295b0332), the private dart repository.

OnePub allows you to privately share Dart packages across your Team and with your customers.

Try it for free and publish your first private package in seconds.

| ![](/files/nLzTOypQkEBrdscFbERX) | <p>Publish a private package in six commands:</p><p><mark style="color:green;"><code>dart pub global activate onepub</code></mark></p><p><mark style="color:green;"><code>onepub login</code></mark></p><p><mark style="color:green;"><code>dcli create --template=full mytool</code></mark></p><p><mark style="color:green;"><code>cd mytool</code></mark></p><p><mark style="color:green;"><code>onepub pub private</code></mark></p><p><mark style="color:green;"><code>dart pub publish</code></mark></p> |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

You can now install `mytool` on any system with dart installed:

```bash
dart pub global activate onepub
onepub login
onepub pub global activate mytool
```

## Overview

The DCli console SDK is intended to to allow you to create Command Line (CLI) Applications from simple scripts to full-blown CLI apps.

DCli is a great replacement for CLI apps that would have traditionally been built with Bash, C, Python, Ruby, Go, Rust ....

Whether it's a 5 line Bash script or a 100,000 line production management system (like we run internally) DCli is the right place to start building your CLI infrastructure.

### So why DCli?

DCli is based on Dart which is a modern programming language that has a set of features that makes building CLI apps easy and reliable.

* Dart and DCli are simple to learn
* Compiled or JIT
* Shebag support (run .dart scripts directly from the terminal ./hellow\.dart)
* Small transportable execs (from 10MB) and the Dart VM is NOT required on the target system.
* Typesafe language catches errors at compile time
* Sound null safety reduces null pointer exceptions
* Fast
* Great development environment using vs-code
* Local and Remote development/debugging
* Cross-platform supporting Linux/Windows/MacOS/ARM

### Example:

```dart
#! /usr/bin/env dcli

import 'dart:io';
import 'package:dcli/dcli.dart';

void main() async {
  var name = ask('name:', required: true, validator: Ask.alpha);
  print('Hello $name');
  
  print('Here is a list of your files');
  find('*').forEach(print);
  
  print('let me copy your files to a temp directory');
  await withTempDirAsync((pathToTempDir) async {
      moveTree(pwd, pathToTempDir);
  });
}
```

To run the above script called hello.dart:

```
./hello.dart
```

### So why is DCli different?

DCli is based on the relatively new programming language; [Dart](https://dart.dev/).

Dart is currently the [fastest growing language on github](https://www.linkedin.com/pulse/google-dart-tops-githubs-list-fastest-growing-2019-bill-detwiler) and is the basis on which Flutter is built.

The [Ubuntu](https://medium.com/flutter/announcing-flutter-linux-alpha-with-canonical-19eb824590a9) installer is now based on Flutter and Flutter will be the primary platform for building GUI's on Ubuntu.

You can now use Dart to build GUI's on Android, IOS, Windows, OSX, Linux. Dart is also suitable for building Web Servers and server-side applications and of course, with DCli you can also target console apps.

Imagine the benefits of using a single language across your complete ecosystem.

Dart is simple to learn and uses the all too familiar 'C' style syntax. I've heard Dart described as the love child of Java and JavaScript. If you come from either of these environments you will find Dart easy to work with.

{% hint style="info" %}
**Dart is the love child of Java and JavaScript and is delightful to work with.**
{% endhint %}

Being easy to learn also helps with the maintenance cycle of your CLI apps. You no longer need a specialized developer, as anyone who has even a vague familiarity with Java, Javascript, or C, ... will be right at home with Dart in a couple of days.

Dart and DCli are easy to install; DCli makes it a breeze to create simple scripts and provides the tools to manage a script that started out as 100 lines but somehow grew to 10,000 lines.

Dart has a large and growing ecosystem of third-party[ libraries](https://pub.dev) that you can include in your CLI app with no more than an import statement and a dependency declaration.

Dart is fast and if you need even more speed it can be compiled into a single file executable that is portable between binary-compatible machines.

```
# compile, install to the local PATH and run hello.dart
$> dcli compile --install hello.dart
$> hello
name: brett
Hello brett

# copy to a remote machine (where dart is NOT installed)
$> scp hello remote.domain.com:

# login to remote machine and run the app hello
$> ssh remote.domain.com
./hello
name: brett
Hello brett
```

You can use your favorite editor to create DCli scripts. Vi or VIM work fine but Visual Code is recommended.

{% hint style="success" %}
**Use Visual Code for the best development experience with Dart.**
{% endhint %}

Visual Code with the `dart-code` extension provides a great development and debugging experience including the ability to develop and debug code on a remote server.


# What does DCli do?

DCli has a singular focus:

{% hint style="info" %}
make it easy to build command line apps using the Dart programming language.
{% endhint %}

DCli has the following aims:

* make building CLI apps as easy as walking.
* fully utilise the expressiveness of Dart.
* works seamlessly with the core Dart libraries.
* provide a cross platform API for Windows, OSx and Linux.
* call any CLI app.
* make debugging CLI apps easy.
* generate error messages that make it easy to resolve problems.
* provide quality documentation and examples.

## DCli's API covers a number of areas:

## User input

Asking the user to input data into a CLI application should be simple. DCli provides a number of functions to facilitate user input.

* ask
* confirm
* menu

```dart
import 'package:dcli/dcli.dart';
void main(){
   var username = ask('Username:', validator: Ask.required);
   if (confirm('Are you over 18')) {
       print(orange('Welcome to the club'));
   }
   
   var selected = menu('Select your poison', options: ['beer', 'wine', 'spirits']
      defaultOption: 'beer');
   print(green('You choice was: $selected'));
}
```

{% hint style="info" %}
**DCli provides an extensive API designed specifically for building command-line apps.**
{% endhint %}

## Displaying information

Out-of-the-box Dart provides the basic 'print' statement which DCli extends to provide common features.

* print
* printerr - prints to stderr.
* colour coding
* cursor management
* clear screen/clear line

```dart
print(orange("I'm an important message"));
printerr(red('Oops, something went wrong here'));
```

## Manage files and directories

A fundamental task of most CLI applications is the management of files and directories. DCli provides a large set of tools for file management such as:

* find
* which
* copy
* copyTree
* move
* moveTree
* delete
* deleteDir
* touch
* exists
* isWritable/isReadable/isExecutable
* isFile
* isLink
* isDirectory
* lastModified

{% hint style="info" %}
Dart has a large ecosystem of packages that you can use to extend the DCli such as the excellent [paths](https://pub.dev/packages/path) package that lets you easily manipulate file paths.
{% endhint %}

```dart
import 'package:dcli/dcli.dart';
void main() {
    if (!exists('/keep')) {
        createDir('/keep');
    }
    
    var images = find('*.png', root: '/images');
    for (var image in images) {
        if (image.startsWith('nsfw')) {
            copy(image, '/keep');
        }
    }
}
```

## Read/Write files

You are often going to need to read/write and modify text and binary files.

* read
* write
* truncate
* append
* replace
* tail
* cat

{% hint style="info" %}
You still have full access to the full set of Dart APIs
{% endhint %}

```dart
//  write lines to myfile.ini
var settings = 'myfile.ini';
settings.write('[section1]');
settings.append('color=red');
settings.append('color=blue');

// fix the spelling in myfile.ini
replace(settings, 'color', 'colour');
```

## Call command line apps

A core feature of DCli is the ability to call other CLI apps and process their output.

* run
* start
* toList
* forEach
* firstLine
* lastLine
* \| (pipe)
* stream

Run mysql with all output being displayed to the console.

```dart
var user = ask('username');
var password = ask('password', hidden:true);
'mysql -u $user --password=$password customerdb -e "select 1"'.run;
```

Run a mysql command and store the results in a list (users).

```dart
var users = 'mysql customerdb -e "select fname, sname from user"'
    .toList(skipLines: 1);
/// now parse each user and print their name
for (var user in users) {
    var parts = user.split(',');
    print('Firstname: ${parts[0]} Surname: ${parts[1]});
}
/// Run grep and print each line that it finds
'grep error /var/lib/syslog'.forEach((line) => print(line));
```

## Explore you environment

DCli makes it easy to explore your environment with direct manipulation of environment variables.

```dart
var user = env['USER'];
env['JAVA_HOME'] = '/usr/lib/java';
```

PATH management tools:

```dart
Env().appendToPath('/home/me/bin');
Env().addToPATHIfAbsent('/home/me/bin');
Env().removeFromPATH('/home/me/bin');
```

And the ability to explore the Dart environment.

```dart
/// Get details of the current dart script that is running.
DartScript.current;
DartSdk().pathToDartExe;
PubCache().pathTo
```

{% hint style="info" %}
You can explore the complete API [here](https://pub.dev/documentation/dcli/latest/).
{% endhint %}


# Install DCli

To get started with DCli you first need to install Dart and optionally the DCli tools.

{% hint style="info" %}
If you just want to use the DCli library then you don't need to install the [DCli tools](/dcli-tools-1/dcli-tools).
{% endhint %}

There are three methods for installing Dart and the DCli tools.

## Option 1) Use DCli library from your project

If you only want to use the DCli library then you can add DCli to your pubspec.yaml as you would any other package.

```yaml
cd /my/dart/project
dart pub add dcli
```

## Option 2) Install Dart/DCli

Start by installing Dart as per:

{% hint style="info" %}
Install Dart from: <https://dart.dev/get-dart>
{% endhint %}

If you want to use the [DCli tools](/dcli-tools-1/dcli-tools), including Shebang (#!) support you need to globally activate DCli.

Now activate the DCli tools:

```
dart pub global activate dcli
dcli install
```

## Option 3) Install Dart/DCli

This is still a work in progress but is intended to provide a three-line script to install Dart and the DCli tools.

{% tabs %}
{% tab title="Linux" %}

```
wget https://github.com/onepub-dev/dcli/releases/download/latest.linux/dcli_install
chmod +x dcli_install
sudo ./dcli_install
```

{% endtab %}

{% tab title="Windows" %}

```
curl https://github.com/onepub-dev/dcli/releases/download/latest.windows/dcli_install.exe
dcli_install.exe
```

{% endtab %}

{% tab title="OSX" %}

```
Coming soon:

curl https://github.com/onepub-dev/dcli/releases/download/latest.osx/dcli_install
dcli_install.exe
```

{% endtab %}
{% endtabs %}

## Install VSCode

You can use virtually any editor to create DCli scripts but we use and recommend Visual Studio Code (vscode) with the Dart-Code plugin.

{% hint style="info" %}
Install Visual Studio Code from: <https://code.visualstudio.com/download>
{% endhint %}

Now install the vs code extension for Dart - Dart Code:

{% hint style="info" %}
Install Dart-Code from: [dartcode.org](https://dartcode.org)
{% endhint %}

We use and recommend the following additional vscode extensions:

* Dart-Code.flutter
* dart-import
* pubspec-assist
* vscode-browser-preview
* bracket-pair-colorizer
* LogFileHighlighter


# Installing on Windows

DCli supports the use of symlinks.

Under Windows to use a symlink you either need to be running as an Administrator or have Windows development mode enabled.

For details on enabling development mode for windows:

{% embed url="<https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development>" %}


# Writing your first CLI app

Let's start by going over the basics of writing the classic Hello World program.

Create a directory to work in:

{% tabs %}
{% tab title="Linux" %}

```bash
mkdir dcli_scripts
cd dcli_scripts
```

{% endtab %}

{% tab title="Windows" %}

```
mkdir dcli_scripts
cd dcli_scripts
```

{% endtab %}

{% tab title="OSx" %}

```
mkdir dcli_scripts
cd dcli_scripts
```

{% endtab %}
{% endtabs %}

Using your preferred editor create a file called 'hello.dart' with following content:

```dart
void main() {
    print('Hello World.');
}
```

{% hint style="info" %}
If you are using vscode you can simply type `code hello.dart` to edit the file.
{% endhint %}

## Running

Now let's run your script:

```dart
dart hello.dart
> Hello World.
```

When we run our script using the `dart` command, Dart performs JIT compilation of our script which slows down the startup time a little but makes for fast test iteration.

{% hint style="info" %}
In vscode you should see 'Run | Debug' just above main(). Click Debug to start your app.
{% endhint %}

## Compiling

You can compile your script to a native executable so that it launches and runs much faster.

{% hint style="info" %}
The [DCli tools](/dcli-tools-1/dcli-tools) allow you to run your app without compiling and without prefixing it with dart.
{% endhint %}

```bash
dart compile exe hello.dart
Generated: hello.exe
./hello.exe
> Hello World

ll hello.exe
-rwxrwxr-x 1 5,875,400 Sep  5 14:30 hello.exe*
```

You now have a completely self-contained executable which you can copy to any binary compatible machine.

The exe is 8MB in size and does NOT require Dart to be installed.

## Dependencies

So far we haven't actually used the DCli API in our hello.dart program. Let's now set up dependency management so we can use the DCli API and any other Dart package.

{% hint style="info" %}
Search [pub.dev](https://pub.dev/) for third party package. You can only use packages labeled 'DART | NATIVE'
{% endhint %}

Dart uses a special file called `pubspec.yaml` to control the set of packages accessible to your application.

Dart's pubspec.yaml is equivalent to a makefile, pom.xml, gradle.build or package.json, in that it defines the set of dependencies for your application.

To use the DCli API or any other Dart package you need to add the dependency to your pubspec.yaml.

Create and edit your first 'pubspec.yaml' file using your preferred editor:

{% hint style="info" %}
Check [pub.dev](https://pub.dev/packages/dcli/install) for the latest version no. of DCli.
{% endhint %}

```bash
name: hello_world
description: My first app that does bugger all.

dependencies:
  dcli: ^7.0.0  # update the version no. to the latest; https://pub.dev/packages/dcli
```

{% hint style="info" %}
The pubspec.yaml lives at the top of your projects directory tree. We refer to this directory as your package root.
{% endhint %}

Whenever you change your 'pubspec.yaml' you must run 'dart pub get' to download the required dependencies:

```bash
dart pub get
Resolving dependencies... 
Got dependencies!
```

## Writing a DCli script

Now that we have added DCli to our pubspec.yaml we can modify hello.dart to make calls to the DCli API.

Edit your hello.dart script as follows:

```bash
import 'package:dcli/dcli.dart';

void main() {
    print("Now let's do someting useful.");

    var username =  ask( 'username:');
    print('username: $username');

    var password = ask( 'password:', hidden: true);
    print('password: $password');

    // create a directory
    if (!exists('tmp')) {
        createDir('tmp');
    }

    // Truncate any existing content
    // of the file 'tmp/text.txt' and write
    // 'Hello world' to the file.
    'tmp/text.txt'.write('Hello world');

    // append 'My second line' to the file 'tmp/text.txt'.
    'tmp/text.txt'.append('My second line');

    // and another append to the same file.
    'tmp/text.txt'.append('My third line');

    // now copy the file tmp/text.txt to second.txt
    copy('tmp/text.txt', 'tmp/second.txt', overwrite: true);

    // lets dump the file we just created to the console
    read('tmp/second.txt').forEach((line) => print(line));

    //let's prove that both files exist by running
    // a recursive find.
    find('*.txt').forEach((file) => print('Found $file'));

    // Now let's tail the file using the OS tail command.
    // DCli uses Dart extensions on the String class allowing us to
    // treat a string
    // as an OS command and `.run` that command as 
    // a child process.
    // Stdout and stderr output are written
    // directly to the console.
    'tail tmp/text.txt'.run;

    //Let's do a word count capturing stdout,
    // stderr will will be swallowed.
    'wc tmp/second.txt'.forEach((line) => print('Captured $line'));

    if (confirm( "Should I delete 'tmp'? (y/n):")) {
        // Now let's clean up
        delete('tmp/text.txt');
        delete('tmp/second.txt');
        deleteDir('tmp');
    }

}
```

Now run our script.

```bash
dart hello.dart
Now let's do someting useful.
username: auser
username: auser
password: *********
password: apassword
Hello world
My second line
My third line
Found /tmp/fred/tmp/second.txt
Found /tmp/fred/tmp/text.txt
Hello world
My second line
My third line
Captured  3  8 41 tmp/second.txt
Should I delete 'tmp'? (y/n): (y/n): y
```

You are now officially a DCli guru.

Go forth young man (or gal) and create.


# Add DCli to your project

If you are just using the DCli API (and not the DCli tools) then adding DCli to your existing app is like adding any other Dart package to your app.

From the CLI

```
cd my/dart/project
dart pub add dcli
```

Now in your Dart code, you can use:

```
import 'package:dcli/dcli.dart';
```

You now have access to the full DCli API.


# pub.dev

DCli is published on [pub.dev](https://pub.dev/packages/dcli) with full details of the [API](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html).


# github

You can find the full source code for DCli on our [github](https://github.com/noojee/dcli) page.


# Dart lambda functions

This section provides details on the Dart language:

To learn more about Dart's syntax read the Dart language tour. <https://dart.dev/guides/language/language-tour>

DCli makes extensive use of Dart's lambdas.

Lambdas are essentially anonymous functions which DCli uses for callbacks.

The most common use of lambdas in DCli are in the forEach method.

In prior examples you have already seen the forEach method in action.

```dart
'tail tmp/nonexistant.txt'.forEach((line) => print(line));
```

The signature of the forEach method is:

```dart
void forEach(LineAction stdout, {LineAction stderr});
```

The first argument `stdout` of the forEach method is a 'positional' argument, the second argument `stderr` is a named argument. The `stdout` argument is required whilst the `stderr` argument is optional.

`LineAction` is a Dart `typedef` that declares that LineAction is a function that takes a single String.

`typedef LineAction = void Function(String line);`

Essentially this means that forEach expects you to pass a function to the first positional argument `stdout` and optionally the second argument `stderr`.

The functions that you pass are unnamed or anonymous functions known as a Lambda.

The following code uses the \[stdout] positional argument to print each line returned by the call to the Linux 'tail' command.

```dart
'tail tmp/nonexistant.txt'.forEach(
    (line) => print(line)
    );
```

The 2nd line is the Lambda function that you provide.

Lets break this down.

The line consists of three components

`(line) => print(line)`

Which can be abstracted to:

`(<args>) => <expression>`

In our example the `stdout` positional argument is of type `LineAction`. The `LineAction` function takes a String as its only argument. So in this case `(<args>)` is a single argument of type String.

What this means is that the `forEach` method will call the Lambda function each time the `tail` command outputs a line. The value of that line will be contained in the `line` argument passed to your Lambda.

The second component is the `=>` operator, sometimes referred to as a 'fat arrow'. Essentially the `=>` operator passes the `line` argument to the `<expression>`;

The third and final component is the `<expression>`.

The `<expression>` can be any valid Dart expression. In the above example the expression is `print(line)` which prints the contents of `line` to the console.

One limitation of the `<expression>` is that it must be a single line expression. Our `LineAction` is declared as returning a void so in our case the return value of the expression is ignored.

Dart's Lambdas actually take two forms; the above 'fat arrow' form and a 'block form'.

The block form allows you to execute multiple statements and requires you to use a `return` statement if you want to return something from the Lambda.

The syntax of the Lambda block form is:

`(args) { <statements> }`

Look carefully and note that the block form doesn't have the 'fat arrow'. I've often been caught when converting a 'fat arrow' form to the block form leaving the 'fat arrow' in place; that just doesn't work.

Block form example of a Lambda:

```dart
'tail tmp/nonexistant.txt'.forEach(
    (line) {
         String trimmed = line.trim();
         print(trimmed);

         // If LineAction took a non-void return type then we could use 
         // a return statement here
         // return 'some value';
    }
);
```

Have a look a the next section of named arguments to understand how to process `stderr` when interacting with the `forEach` method.


# Function Arguments

This section provides details on the Dart language:

To learn more about Dart's syntax read the Dart language tour. <https://dart.dev/guides/language/language-tour>

Dart allows three types of arguments to be passed to a method or function.

Positional, optional and named.

Positional arguments are traditional 'C' style arguments that everyone is familiar with.

Optional arguments are positional arguments that are (you guessed it) optional.

Named arguments are a little trickier but depending on your current language experience you may already be familiar with them.

Named arguments allow you to pass an argument by name rather than position and they are (usually) optional.

To declare a named argument you use curly braces `{}`.

```dart
void testMethod(String arg1, [String arg2], {String? arg3, String? arg4});
```

In the above example `arg1` is a positional argument, `arg2` is an optional argument with `arg3` and `arg4` being optional named arguments.

To call the test method passing values to all of the above arguments you would use:

```dart
testMethod('value1', 'value2' , arg4: 'value4', arg3: 'value3');
```

Note that I've reversed the order of `arg3` and `arg4`. As they are named arguments you can place them in any position AFTER the named and optional arguments.

Let's finish with an example using the forEach method:

```dart
'tail /var/log/syslog'.forEach((line) => print(line), stderr:(line) => print(line));
```

The above example will print any output sent to stdout or stderr from the 'tail' command.


# Futures

## DCli and futures

if your not a Dart programmer (yet) one of the most difficult things about Dart are Futures. If you are familiar with Javascript then a Future is the equivalent of a Promise.

### Just ignore Futures

If your not familiar with Dart or Javascript then for the moment you can just ignore futures.

DCli works very hard to ensure that you don't need to worry about Futures.

This is very intentional.

If you stick to using DCli's built in functions then you can completely ignore Futures. If you start importing Dart's core libraries or third party libraries then you need to pay attention to return types.

The first time you try to call a method or function that returns a `Future` then you will know its time to come back here and read about Futures.

Until then, you can just skip this section.

### How DCli manages futures

DCli does not stop you from using `await`, `Futures`, `Isolates` or any other Dart functionality. It's all yours to use and abuse as you will.

DClis global functions however intentionally avoid `Futures`.

The aim of DCli is to create a Bash-like simplicity to building CLI apps. `Futures` are great and all but they do make the code more complex and harder to read.

Futures also can make your scripts a little dangerous. If you copy a file and then want to append to the copied file, you had better be certain that the copy command has been completed before you start the append. DCli's global functions remove those complications.

If you are interested in how we avoid using `Futures` read up on `waitFor` and check out DCli's own `waitForEx` function that does stacktrace repair when an exception is thrown.

When you need to use futures you can read up on them in the Dart language Tour:

<https://dart.dev/guides/language/language-tour>


# stdin/stdout/stderr a primer

When building console apps you are going to hear a lot about three little entities: stdin, stdout and stderr.

In the Linux, Windows and OSX world any time you launch an application three file descriptors are automatically opened and attached to the application.

I refer to these three file descriptors as 'the holy trinity'. If you are going to do Command Line Interface (CLI) programming then it is imperative that you understand what they are and how to use them.

This primer discusses the origins, the structure and finally how to interact with the holy trinity in CLI apps.

{% hint style="info" %}
stdin/stdout/stderr are not unique to dart. Virtually every langue and OS supports them.
{% endhint %}

At the most basic level, each of these files is intended to have a specific purpose.

stdin - lets you read input from a user

stdout - lets you show information to the user

stderr - lets you show errors to the user.

As a visual aid, you can think of the files as:

```
[stdin -> app -> stdout
              -> stderr]
```

But these files are anything but basic and it's not just a user that we can communicate with.

## In the beginning

Let's take a little history lesson.

Way back in the dark ages (circa 1970) the computer gods got together and created Unix.

{% hint style="info" %}
And Dennis said, let there be 'C'. And Denis looked upon 'C' and said it was good and the people agreed.

But Dennis did not rest on the seventh day, instead, he called upon Kenneth and over lunch and a nice red, they doth created Unix.

Dennis Ritchie; 9th Sept 1944 - 12th Oct 2011\
Kenneth Lane Thompson February 4, 1943
{% endhint %}

![My first bible.](/files/-MH41E6m8oxfg6m8ghcw)

Unix is the direct ancestor of Linux, MacOS and to a lesser extent Windows. You might more correctly say that 'C' is the common ancestor of all three OSs, as their kernels are all written in C.

As C was taken up as the primary language for writing Operating Systems, the concept of stdin/stdout and stderr proliferated across the OS world.

The result is today that a large no. of operating systems support stdin/stdout and stderr.

The majority of people reading this primer will be working with Linux, MacOS or Windows and in each of these cases, the Holy Trinity (stdin/stdout/stderr) is available in every app they use or write.

The following examples are presented using the Dart programming language, but the concepts and even most of the details are correct across multiple OSs and languages.

## When you have a hammer, everything's a snail

In the Unix world, EVERYTHING is a file. Even devices and processes are treated as files.

{% hint style="info" %}
If you know where to look, processes and devices are visible in the Linux/MacOS directory tree as files.
{% endhint %}

So if everything is a file, does that mean we can directly read/write to a device/process/directory?

The simple answer is, yes.

If we want to read/write to a file we need to open the file. In the Unix world (and virtually every other OS) when we open a file we get a 'file descriptor' or FD for short. Once we have an FD we can read and write to the file. The FD may be presented differently in your language of choice but under the hood, it's still an FD. (**In Dart we have the File class that wraps an FD**).

> The terms 'file descriptor' and 'file handle' are often used interchangeably.

So what exactly is an FD? Under the hood, an FD is just an integer that acts as an index to an array of open files. The FD array contains information such as the path to the file, the size of the file, the current seek position and more.

## The Holy Trinity

So now we understand that in Unix everything is a file, you probably won't be surprised when I tell you that stdin/stdout/stderr are also files.

So if stdin/stdout/stderr are files, how do you open them?

The answer is you don't need to open them, the OS opens them for you. When your app starts, it is passed one file descriptor (FD) for each of stdin/stdout/stderr.

If you recall, we said that an FD is just an integer, indexing into an array of structures, with one array entry for each open file. Each application has its own array. When your app starts that array already has three entries, stdin, stdout and stderr.

The order of those entries in the array is important.

\[0] = stdin

\[1] = stdout

\[2] = stderr.

If you open any additional files they will appear as element \[3] and greater.

## The tower of Babel

If you have done any Bash, Zsh, Command or Powershell programming you may have seen a line similar to:

```
find . '*.png' >out.txt 2>&1
```

You can't get much more obtuse than the above line, but now we know about FD's it actually makes a little more sense.

{% hint style="warning" %}
Bash was not created by the gods. I think the other bloke had a hand in this one.
{% endhint %}

The `>out.txt` section is actually a shorthand for `1>out.txt` . It instructs Bash to take anything that `find` writes to FD =1 (stdout) and re-write it to the file called 'out.txt'.

The `2> &1` section instructs Bash to take anything `find` writes to FD=2 (stderr) and re-write it to FD=1.

i.e. anything written to stderr (FD=2) is re-written to stdout (FD=1) which in turn is written to `out.txt`.

The result of the above command is that both stdout and stderr are written to the file called 'out.txt'.

It would have been less obtuse to write:

```
find . '*.png' 1>out.txt 2>out.txt
```

But of course, we are talking about Bash here and apparently more obtuse is always better :D

> Many other shells use a similar syntax.

Most languages provide a specific wrapper for each f these file handles. In Dart we have the global properties:

* stdin
* stdout
* stderr

> The 'C' programming language has the same three properties and many other languages use the same names.

## And on this rock, I will build my app

I like to think of the Unix philosophy as programming by Lego (but Meccano is superior).

{% hint style="info" %}
Unix was all about Lego - build lots of little bricks (apps) that can be connected.
{% endhint %}

In the Unix world (and the Dart world) every CLI app you write contributes to the set of available Lego bricks. But Lego bricks would be useless unless you can connect them. In order to connect bricks the 'pegs' on each brick must match the 'holes' on other bricks and that's where stdin/stdout/stderr come in.

In the Unix world every brick (app) has three connection points:

* stdin - a hole for input
* stdout - a peg for normal output
* stderr - a peg for error output

Any peg can go into any hole.

You might now have guessed that you can connect stdout from one program to stdin on another program:

\[myapp -> stdout] => \[stdin -> yourapp]

If you are familiar with Bash you may have even seen one of the common ways to connect two apps.

```
ls "*.png" | grep "turtles"
```

In the above example, the `ls` command will write a list of files that end in `.png`. The grep command will receive that list and then output a line each time it sees a filename that contains the word `turtles`.

The Bash '|' pipe operator connects the stdout of 'ls' to the stdin of 'grep'.

If you like, the 'pipe' command is the plumbing and Bash is the plumber.

Any data `ls` writes to it's stdout, is written to 'grep's stdin. We say that the two apps are connected via a 'pipe'.

> **A 'pipe' is just a program that reads from one FD and writes to another.**
>
> In this case Bash is acting as the pipe. When Bash sees the '|' character it takes it as an instruction to launch the two applications (ls and grep), read stdout from ls and write that data to grep's stdin.

A couple of other interesting things happened here.

1\) stdin of `ls` is still connected to the terminal (`ls` is just ignoring it)

2\) stdout of `grep` is still connected to the terminal, anything that grep writes to its stdout will appear on the terminal.

## Revelations

{% hint style="warning" %}
You take the *red* pill—you stay in Wonderland, and I show you how deep the rabbit hole goes.
{% endhint %}

So let's just stop for a moment and consider this fact; **the terminal you are using is actually an app!**

Like every other app, it has stdin/stdout/stderr.

When we run an app in a terminal window the app's:

* stdin is attached to the terminal's stdout
* stdout is attached to the terminal's stdin.
* stderr is attached to the terminal's stdin.

\[terminal -> stdout] => \[stdin -> app -> stdout, stderr] => \[stdin -> terminal]

{% hint style="info" %}
And so we are all connected in the great Circle of Life.

Mafasa, The Lion King.
{% endhint %}

So let's look at what happens when our app prints something.

> \[print('hello') -> stdout] => \[stdin -> terminal font] => \[graphics card ] => \[eye -> brain]

When we call `print('hello')` our app writes 'hello' to stdout, this arrives in the terminal app via the terminal's stdin.

The terminal app then takes the ASCII characters we sent (hello), translates them to pixels and sends them to our graphics card.

These pixel form, what many people like to call, a 'font'. Somehow, rather magically, your brain translates these little pixels into characters and you see the word 'hello'.

{% hint style="info" %}
In the beginning, was the Word, and the Word was 'hello world'.
{% endhint %}

The above example uses `print` to write to stdout. Print is a common function for writing to stdout and `print` or similar exists in most languages. Under the hood `print` literally writes to stdout:

If we look at the Dart implementation of `print,` the truth of this is self evident.

```
void print(String message)
{
    stdout.writeln(message);
}
```

{% hint style="info" %}
And you will know the truth, and the truth will set you free.

Some bloke.
{% endhint %}

## It's turtles, all the way down!

So I lied. But it was an honest lie...

Launching a terminal doesn't directly attach to our app as there is almost always a middleman. That middleman is the shell.

The shell, as I'm sure you know, provides an interactive prompt allowing you to launch applications.

So my (small) lie can be fixed by adding the shell into the pipeline:

Instead of:

\[print('hello') -> stdout] => \[stdin -> terminal -> font] => \[graphics card ] => \[eye -> brain]

What really happens is:

\[print('hello') -> stdout]

\=> \[stdin -> shell -> stdout]

\=> \[stdin -> terminal -> font]

\=> \[graphics card ]

\=> \[eye -> brain]

Examples of shells are:

Bash, Zsh, Powershell, CMD, Ash, Bourne, Korn, Hamilton...

and of course, you could build your own.

{% hint style="info" %}
Is that a rhetorical point, or would you like to do the maths?

Sheldon's Mother
{% endhint %}

OK, so let's do the maths and implement a basic shell.

You can find the complete code on github: <https://github.com/onepub-dev/dshell>

#### dshell.dart

```dart
#! /usr/bin/env dcli

import 'dart:io';

import 'package:dcli/dcli.dart';
import 'package:dshell/src/app_with_args.dart';
import 'package:dshell/src/pipe.dart';

void main(List<String> args) async {
  // Loop, asking for user input and evaluate it
  for (;;) {
    // print pwd> as a prompt
    stdout.write('${green(basename(pwd))}${blue('>')}');
    final commandLine = stdin.readLineSync() ?? '';
    if (commandLine.isNotEmpty) {
      await evaluate(commandLine);
    }
  }
}

// Evaluate the user's input
Future<void> evaluate(String commandLine) async {
  // use the | to split out multiple commands
  final apps = commandLine.split('|');
  // just a single app, so run it.
  if (apps.length == 1) {
    runApp(AppWithArgs(apps[0]));
    return;
  }
  // if we see two apps use pipe 
  if (apps.length == 2) {
    final app1 = AppWithArgs(apps[0]);
    final app2 = AppWithArgs(apps[1]);

    await simplePipe(app1, app2);
  } else {
    stderr.writeln('We only support piping 2 apps');
  }
}

void runApp(AppWithArgs appWithArgs) {
  switch (appWithArgs.app) {
    // list files in the current directory
    case 'ls':
      ls(appWithArgs.args);
      break;

    // change directories
    case 'cd':
      Directory.current = join(pwd, appWithArgs.args[0]);
      break;

    // treat the first word as the name of an app
    // and run it.
    default:
      if (which(appWithArgs.app).found) {
        // The run command is part of DCli and does all of the
        // plumbing for stding/stdout/stderr.
        run(appWithArgs.cmdLine);
      } else {
        stdout.writeln(red('Unknown command: ${appWithArgs.app}'));
      }
      break;
  }
}

/// our own implementation of the 'ls' command.
void ls(List<String> patterns) {
  if (patterns.isEmpty) {
    find('*',
            workingDirectory: pwd,
            recursive: false,
            types: [Find.file, Find.directory])
        .forEach((file) => stdout.writeln('  $file'));
  } else {
    for (final pattern in patterns) {
      find(pattern,
              workingDirectory: pwd,
              recursive: false,
              types: [Find.file, Find.directory])
          .forEach((file) => stdout.writeln('  $file'));
    }
  }
}


```

#### `pipe.dart`

The pipe function is where the funky stuff happens.

The simplePipe function runs each app and then wires their output together using dart's built-in pipe command. The pipe command simply reads `stdout` of the first app and writes that data into `stdin` of the second app.

\[app1 -> stdout] => \[stdin -> app2]

Finally, the call to pipeNoClose wires the output of the app2 is written directly to our shell's own `stdout`.

\[app1 -> stdout] => \[stdin -> app2] => \[stdout(shell)] => \[stdin -> terminal ....] => brain

The result is, that the data that app2 writes is displayed on the console (because the console is reading the shell's stdout).

This is essentially the same process used by any shell.

```dart
import 'dart:io';

import 'app_with_args.dart';

Future<void> simplePipe(AppWithArgs app1, AppWithArgs app2) async {
  final app1Process = await Process.start(app1.app, app1.args);
  final app2Process = await Process.start(app2.app, app2.args);

  // the output from app1 is sent to the input of app2
  await app1Process.stdout.pipe(app2Process.stdin).catchError(
    // ignore: avoid_types_on_closure_parameters
    (Object e) {
      // ignore broken pipe after app2 process exit
    },
    test: (e) =>
        e is SocketException &&
        (e.osError!.message == 'Broken pipe' ||
            e.osError!.message == 'StreamSink is closed'),
  );

  /// the output of app2 is sent to the console.
  /// We can't use the normal pipe command is it closes the consumer (stdout)
  /// would would stop our app from outputting any further
  await pipeNoClose(app2Process.stdout, stdout);
}

Future<void> pipeNoClose(Stream<List<int>> stdout, IOSink stdin) async {
  await stdin.addStream(stdout);
}

```

If you clone and run the above Dart script, you get an interactive shell. Here is a sample session:

```bash
git clone https://github.com/onepub-dev/dshell.git
cd dshell
dart bin/dshell.dart 
example> ls
  dshell.dart
example> mkdir tmp
example> cd tmp
tmp> touch me
tmp> ls
  me
tmp> cd ..
example> ls
  dshell.dart
  tmp
example> cat bin/dshell.dart | grep pipe
   import 'package:dshell/src/pipe.dart';
   await pipe(app1, app2);
```

## And a word from our sponsors

This Blog and DCli are sponsored by [OnePub](https://onepub.dev/drive/3aacb2de-3eb5-4cc5-90f3-60347aa2dc11).

OnePub is a private package repository for Dart.

If you want to try OnePub, you can publish our sample shell application in a few lines:

```bash
dart pub global activate onepub
onepub login
git clone https://github.com/onepub-dev/dshell.git
cd dshell
onepub pub private
dart pub publish
```

You can now install your own shell anywhere you have Dart.

```bash
onepub pub global activate dshell
```

OnePub is currently in beta (as of Aug 2022). Whilst in Beta, anyone that publishes a package to OnePub will receive a free lifetime subscription.

### Its turtles all the way down

So let's look at what actually happens when you launch a terminal window or connect to a console.

When the terminal window launches it creates a canvas to display text and starts listening to keystrokes. If the terminal window has the focus then the OS will send keystrokes to it, otherwise, it gets nothing. The terminal launches your default shell as a child process. Let's call this shell `Bash` but it could be called `Powershell`.

When Bash is launched, it, like every other app, receives three file descriptors stdin/stdout and stderr.

The terminal window, being an app, also has its own stdin/stdout and stderr.

When we launch a CLI app its stdin is attached to the Terminal (via the shell).

It's actually the terminal app that is responsible for interacting with the keyboard.

When the terminal app gains focus it is attached to the system message queue (allowing it to receive keystrokes) and the terminal app writes characters to our CLI app's stdin (via the shell).

\[brain -> fingers] -> \[keyboard -> system queue] -> \[terminal app] -> \[shell] -> \[stdin of our CLI app]

### Stdin

Let's recap.

* Stdin allows an app to take input from the user or another app.
* Because it's a standard, tools like Bash can reliably use it to wire apps together.
* You can't assume that your app's stdin is only taking data from the keyboard it could be another app.
* Many apps provide an interactive and non-interactive mode to cater for the different ways that it can be launched.
* This doesn't mean that you have to handle data coming from a user or another app. If those modes don't suit the purpose of your app you can just ignore stdin.
* Whilst not discussed here, stdin usually operates in line mode with the shell echoing all typed characters to the console. In most languages, you can switch off echo mode (for password capture etc) as well as switching to non-line mode.
* You can't use a 'seek' on stdin to change the file read position.

{% hint style="info" %}
In DCli we use the `ask` function which provides a high-level wrapper for readLineSync.

var name = ask('Enter your name:');
{% endhint %}

## Stdout

Most languages provide a **print** and often a **println** function, both of which write to stdout.

Normally, print will print without a terminating newline, whilst println includes a terminating newline.

In Dart, we only have the print function (which includes a terminating newline) but DCli adds an 'echo' function that allows you to control if a newline is added.

You can of course write directly to stdout.

## Stderr

Most languages don't provide a method to easily write to stderr. You will generally need to write something like:

`stderr.write('bad times, where had by all');`

The DCli package adds the `printerr` function which works exactly like print does, but prints to stderr.

## Conclusion

Well, that was quite a trip. Hopefully, it fills some gaps and puts you on a path to building better CLI tooling.

The OnePub Blog - [The Dart Side](https://onepub.dev/drive/9a6ed12b-5ae2-4299-b182-e97f078dd689) has additional articles on CLI programming


# Overview

The DCli API provides an extensive set of functions all focused on building CLI apps with Dart.

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

DCli exposes a significant no. of global functions and most of your interaction with the DCli API will be via these global functions.

DCli also exposes a number of Dart Classes which generally provide more specialized functionality.


# Using DCli functions

## Using DCli functions

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

Let's start by looking at some of the built-in functions that DCli supports.

DCli exposes a range of built-in functions that are exposed as Dart global functions.

These functions are the core of how DCli provides a very Bash-like feel to writing DCli scripts.

These functions make strong use of named arguments with intelligent defaults so mostly you can use the minimal form of the function.

Take note, there are no `Futures` or `await`s here. Each function runs synchronously.

```dart
import 'package:path/path.dart';
import 'package:dcli/dcli.dart';

void main() {
    // Use the global DCli Settings to enable debug output.
    Settings().setVerbose(enabled: true);

    // Print the current working directory
    print('PWD: ${pwd}');

    // Create a directory and if necessary
    // its parent directories.
    var pathToImages = 'tools/images';
    createDir(pathToImages, recursive: true);

  
    var pathToGoodJpg = join(pathToImages, 'good.jpg');
    // create a file (it's empty)
    touch(pathToGoodJpg, create: true);

    // update the last modified time on an existing file
    touch(pathToGoodJpg);

    print('Showing all files');

    // print out all files in the current directory.
    // [file] is just a [String]
    find('*.*', recursive: false).forEach((file) => print(file));

    // take a nap for a couple of seconds.
    sleep(2);

    print('Find file matching *.jpg');
    // Find all files that end with .jpg
    // in the current directory and any subdirectories
    for (var file in find('*.jpg', workingDirectory: pathToImages).toList()) {
        print(file);
    }

    var pathToBadJpg = join(pathToImages, "bad.jpg");
    // Move/rename a file
    move(pathToGoodJpg, pathToBadJpg);

    // check if a file exists.
    if (exists(pathToBadJpg)) {
        print("bad.jpg exists");
    }

    // Delete a file asking the user first.
    delete(pathToBadJpg, ask: true);

}
```

As you can see we have achieved much of the power of Bash without any of the ugly grammar, and what's more we only used one type of quote!


# User input

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

Asking the user to input data into a CLI application should be simple. DCli provide a number of core methods to facilitate user input.

* ask
* confirm
* menu

## Ask

The 'ask' function provides a simple but flexible means of requesting information from the user.

In its simplest form, you can ask the user for an input string.

{% hint style="info" %}
Any user input **whitespace** is stripped before it is validated or returned.
{% endhint %}

### Arguments

#### prompt

The prompt is the only positional argument that `ask` takes. Ask will display the prompt verbatim.

```dart
var username = ask('Username:');
```

```bash
Username: brett
```

You may pass an empty string for the prompt in which case no prompt will be displayed.

#### hidden

You can request that the user input isn't echoed back to the user:

```dart
var password = ask('Password:', hidden: true);
```

```bash
Password: ******
```

#### defaultValue

You can provide a default value. If the user hits enter without entering any text then the default value will be returned.

```dart
var username = ask('Username:', defaultValue: 'Administrator');
```

```bash
Username: [Administrator]
```

If you combine a defaultValue with the hidden argument then the default value will be rendered as 6 '\*'.

```dart
var password = ask('Password:', hidden: true, defaultValue: 'a secret');
```

```bash
password: [******] 
```

If you combine a defaultValue with an empty prompt then Ask will not display the prompt nor the default value.

```dart
var secretQuestion = ask('', defaultValue: 'a secret', required: false);
```

See the 'customPrompt' argument to modify how the default is displayed.

#### validator

Ask takes a validator. If the entered input doesn't match the supplied validator then the user will be re-prompted until they enter a valid value.

```dart
var age = ask('Age:', validator: Ask.integer);
```

```bash
Age: abc
Invalid integer.
Age:
```

See the section on [validators](/dcli-api/user-input/ask-validators) for more details.

#### required

By default, the ask function requires the user to enter a non-blank line (whitespace is stripped from user input before it is evaluated).

If you want to make a user value optional either pass in a defaultValue or pass required: false

```dart
var age = ask('Age:', required: false, validator: Ask.integer);
```

#### customPrompt

Since: 2.0.0

By default when passing a default value the `ask` command formats the default within brackets:

```dart
var username = ask('Username:', defaultValue: 'Administrator');
```

```bash
Username: [Administrator]
```

You can completely modify the prompt by providing the `customPrompt` argument.

```
final response = ask('say something:', defaultValue: 'my default'
    , customPrompt: (prompt, defaultValue, hidden) { 
      if (hidden) { 
        return '$prompt>'; 
      } 
      else { 
        return '($defaultValue) $prompt>'; 
      } 
  });
```

Be careful to suppress displaying the default value when `hidden` is true, otherwise, you may end up displaying a password.

## Confirm

The confirm method allows you to ask the user for a true/false response and returns a bool reflecting what the user entered.

```dart
bool allowed = confirm('Are you over 18', defaultValue: false);
```

### Arguments

#### prompt

The prompt is the only positional argument that `confirm` takes. Confirm will display the prompt verbatim.

```dart
var alive = confirm('Are you alive:');
```

```bash
Are you alive (y/n): y
```

You may pass an empty string for the prompt in which case no prompt will be displayed.

#### defaultValue

You can provide a default value. If the user hits enter without entering any text then the default value will be returned.

```dart
var confirmed = confirm('Are you sure:', defaultValue: true);
```

The default value is capitalized.

```bash
Are you sure: (Y/n):
```

#### customPrompt

Since: 2.0.0

By default when passing a default value the `confirm` command formats the default within brackets:

```bash
Are you alive (y/n): y
```

You can completely modify the prompt by providing the `customPrompt` argument.

```
  final confirmed = confirm('Are you sure?', defaultValue: false,
      customPrompt: (prompt, defaultValue) {
    var yes = 'yes';
    var no = 'no';

    if (defaultValue != null) {
      yes = defaultValue ? 'Yes' : 'yes';

      no = !defaultValue ? 'No' : 'no';
    }
    return '$prompt> [$yes/$no]';
  });
```

Are you sure?> \[yes/No]

## Menu

The menu function allows you to display a list of menu items for the user to select from and returns the selected item.

```dart
var selected = menu('Select your poison'
   , options: ['beer', 'wine', 'spirits']
   , defaultOption: 'beer');
print(green('You chose $selected'));
```

```
1) beer
2) wine
3) spirits
Select your poison: 1
```

You can also specify a default option. If you pass a default value and the user hits enter without entering a value then the default value will be returned.

The list of options can be a String or a Dart class. By default menu will call toString on any object passed but you can pass the format argument to control how each option is displayed:

```dart
class Car
{
   String make;
   String model;
   Car(this.make, this.model);
}
var available = [Car('Ford', 'Falcon'), Car('Holden', 'Capree'), Car('BMW', 'M3')];
var selected = menu('Choose your preferred car:'
   , options: available
   , format: (Car car) => '${car.make} ${car.model}'
   , defaultOption: available[0]);
print(green('You chose $selected'));
```

```
 1) Ford Falcon
 2) Holden Capree
 3) BMW M3
Choose your preferred car: [1] 
```

### Arguments

#### options

A list of options for the user to select from.

The list can be a list of Strings or a list of Dart objects (all of the same type).

#### defaultOption

Specifies the defaultOption from the list of `options`. The `defaultOption` must be of the same type as the items in the `options` list.

The default option will be colored-coded in the list if your terminal supports ANSI escape codes.

The default option will be displayed as an index after the prompt.

#### format

By default the menu function will display each option by calling `toString` on the passed option.

You can provide an alternate formatter for each option by passing a lambda to the format argument.

```dart
format: (Car car) => '${car.make} ${car.model}'
```

#### customPrompt

Since: 2.0.0

The customPrompt allows you to modify the selection prompt.

```dart
customPrompt: (prompt, defaultOption) {
  return '$prompt> $defaultValue';
}
```

#### limit

If you pass in a large `option` list you can pass in the `limit` argument to limit the number of options displayed in the menu. The first `limit` options in the list of options will be displayed.

#### fromStart

FromStart is true by default. If you set it to false and you pass a `limit` then the menu will show the last `limit` options.


# Ask Validators

## Overview

DCli ships with a number built in validators for use with the ask function.

When a validator is applied to the ask method, the ask method will not return until the user enters a value that satisfies the validator.

{% hint style="warning" %}
If you pass required: false to`ask`, then the validator won't be called if the user input is empty!
{% endhint %}

In addition to the built-in validators, you can also create your own custom validators and combine multiple validators.

## Combining Validators

The DCli Ask command allows you to combine multiple validators with the Ask.any and Ask.all validators.

### Ask.all

The Ask.all validator takes an array of validators.

All validators must succeed for the input to be considered valid. The validators are processed in the order they are passed (left to right). The error from the first validator that fails is displayed.

The Ask.all validator is the equivalent of a boolean AND operator.

It should be noted that the user input is passed to each validator in turn and each validator has the opportunity to modify the input. As a result, each validator will be operating on a version of the input that has been processed by all validators that appear earlier in the list.

```dart
 var password = ask( 'Password?', hidden: true
      , validator: Ask.all([Ask.alphaNumeric, AskLength(10,16)]));
```

The password must be composed of alphanumeric characters **and** be between 10 and 16 characters long.

### Ask.any

The Ask.any validator takes an array of validators.

Only one of the validators must succeed for the input to be considered valid. The validators are processed in the order they are passed (left to right). If no validators pass, then the error from the first validator is displayed.

The Ask.any validator is the equivalent of a boolean OR operator.

It should be noted that the user input is passed to each validator in turn and each validator has the opportunity to modify the input. As a result, each validator will be operating on a version of the input that has been processed by all validators that appear earlier in the list.

If none of the validators pass then the error from the first validator that failed is displayed. The implication is that the user will only ever see the error from the first validator.

```dart
 var password = ask( 'Password?', hidden: true
      , validator: Ask.all([Ask.alphaNumeric, AskValidatorLength(10,16)]));
```

## Standard Ask Validators

The set of standard Ask Validators allows you to validate common input requirements. You can combine them with the Ask.all and Ask.any methods to allow for more complicated validation.

All of the standard Ask Validators allow the user to enter a blank value

{% hint style="info" %}
If you only want the validator applied if a user enters a value, then pass 'required: false' to the ask function.
{% endhint %}

### Ask.ipAddress

Validates that the entered value is an IP address. By default, both IPv4 and IPv6 addresses are permitted.

To restrict the IP address to a specific version, pass in the expected version.

```dart
var ipAddress = ask( 'Server IP?',  validator: Ask.ipAddress());
var ipV4Address = ask( 'Merchant IP?'
    , validator: Ask.ipAddress(AskValidatorIPAddress.ipv4));
```

### Ask.lengthMax

Validates that the input is no longer than the provided maximum.

```dart
var username = ask( 'username?', validator: Ask.lengthMax(32));
```

### Ask.lengthMin

Validates that the input is no shorter than the provided minimum.

```dart
var username = ask( 'username?', validator: Ask.lengthMin(26));
```

### Ask.lengthRange

Validates that the input is no shorter than the provided minimum and no longer than the provided max.

```dart
var username = ask( 'username?', validator: Ask.lengthRange(26, 32));
```

### Ask.inList

Validates that the input is contained in the provided list.

{% hint style="info" %}
You are often better off using a menu.
{% endhint %}

Set the caseSensitive to true to do a case-sensitive comparison against the list. Defaults to false.

The list may contain strings or any Dart Object.

The toString method is called on each object passed to the list to obtain the comparison string.

```dart
var sex = ask( 'sex?', validator: Ask.inList(['male', 'female']));
```

### Ask.email

Validates that the user input is a valid email address.

```dart
var email = ask( 'Email Address?', validator: Ask.email));
```

### Ask.fqdn

Validates that the user input is a valid Fully Qualified Domain Name ([www.onepub.dev](http://www.onepub.dev)) address.

```dart
var email = ask( 'FQDN?', validator: Ask.fqdn));
```

### Ask.integer

Validates that the user input is a valid integer.

The integer is returned as a string.

```dart
var ageAsString = ask( 'Age?', validator: Ask.integer));
var age = int.parse(ageAsString);
```

### Ask.valueRange

Validates that an entered number is within the provided range (inclusive). Can be used with both integer and decimal no.s

The value is returned as a string.

```dart
var age = ask('Age?', 
    validator: Ask.all([Ask.integer, Ask.valueRange(18, 25)]));
```

### Ask.decimal

Validates that the user input is a valid decimal number.

The decimal is returned as a string.

```dart
var age = ask( 'Age?', validator: Ask.decimal));
```

### Ask.alpha

Validates that the user input is an alpha string with every character in the range \[a-zA-Z].

```dart
var name = ask( 'name?', validator: Ask.alpha));
```

### Ask.alphaNumeric

Validates that the user input is a alphaNumeric string with every character in the range \[a-zA-Z0-9].

```dart
var name = ask( 'name?', validator: Ask.alpha));
```

## Custom Validators

You can also write your own validators.

All validators must inherit from the AskValidator class and implement the validate method.

The validator method must return the passed line but may alter the line before returning it. The altered results are what will be returned from the ask function.

{% hint style="warning" %}
a validator MUST not include the value of the 'line' in an error message as you risk exposing a password that the user is entering.
{% endhint %}

If the ask function uses one of the combination validators (Ask.all, Ask.any) then the line input by the user will be passed to each validator in turn. Each validator may change the line and that altered value will be passed to the next validator. In this way, the entered value may go through multiple transformations before being returned to the caller.

```dart
class AskGoodOrBad extends AskValidator {
  const AskGoodOrBad();
  @override
  String validate(String line) {
    line = line.trim();

    if (line != 'good' && line != 'bad') {
      throw AskValidatorException(red('The response must be good | bad'));
    }
    return line;
  }
}
```

To use your your new validator:

```dart
var getsPresent = ask('Have you been good or bad'
    , validator:  AskGoodOrBad());
```

### Async Validators

An Ask validator must return a synchronous type. If you need to make an async call from within validator then you need to use the waitForEx function to strip the async nature of the call.

```dart
class AskAsync extends AskValidator {
  const AskAsync ();
  @override
  String validate(String line) {
    line = line.trim();
    
    if (checkInput(line) == false)
    {
      throw AskValidatorException(red("The entered line wasn't valid"));
    }
    return line;
  }
  
  Future<bool> checkInput(String line) async {
    // make some async call to check [line]
  }
}
```


# Displaying information

DCli provides a number of methods to display information to a user:

## print

Dart provides the built in function 'print' which prints a line of text including a new line.

```
print('hello world');
```

## printerr

The standard Dart 'print' function prints to stdout, DCli's 'printerr' function is identical except that it prints to stderr.

```
printerr('something bad happened.');
```

You should use printerr when you are printing error messages.

## echo

The echo function is provided to supplement the Dart print method. The 'echo' allows you to control whether a new line is output after the text. By default echo will NOT output a newline.

```dart
echo('hello', newline: false);
```

## Colour coding

DCli allows you colour code your text output.

```dart
print(orange('hello world'));
```

You can also control the background colour:

```dart
print(orange('hello world', background: AnsiColor.white));
```

The following colours are supported for both the foreground (text) and background colours.

* red
* black
* green
* blue
* yellow
* magenta
* cyan
* white
* orange
* grey

By default the bold attribute is attached to each of the above builtin colours. You can suppress the bold attribute:

```
 print(red('a dark message', bold: false));
```

## Format().row

This method is considered experimental. Use at your own peril.

The row method allows you to output a row of fixed with columns with controlled alignment.

```
print(Format().row(['OS Version', '${Platform.operatingSystemVersion}'],
        widths: [17, -1]));
```

Outputs a row with two columns. The first is 17 characters wide, the second expands as needed.

```
print(Format().row([
          '$label',
          '${fstat.modeString()}',
          '<user>:${(owner.group == owner.user ? '<user>' : owner.group)}',
          '${privatePath(path)} '
        ], widths: [
          17,
          9,
          16,
          -1
        ], alignments: [
          TableAlignment.left,
          TableAlignment.left,
          TableAlignment.middle,
          TableAlignment.left
        ]));
```

Outputs a row with four columns of widths 17, 9, 16 and infinite. The columns are aligned, left, right, middle and left.

## clearScreen

Clears the console.

```
clearScreen();
```

## clearLine

Clears the current line.

```
clearLine();
```

### writeLine

Writes \[text] to the console followed by a newline.

You can control the alignment of \[text] by passing the optional \[alignment] argument which defaults to left alignment. The alignment is based on the current terminals width with spaces inserted to the left of the string to facilitate the alignment. Make certain the current line is clear and the cursor is at column 0 before calling this method otherwise the alignment will not work as expected.

## Cursors

Ansi terminals support the concept of a cursor.

Characters printed to the terminal a displayed in a grid of rows and columns.

Historically terminals were generally 24 rows x 80 columns but modern terminals can be any size.

The number of rows and columns is determined by the size of the terminal window.

A cursor describes a location on the terminal within the grid.

You can move the cursor to any location and then print text at that location.

Cursors allow you to build advanced user interfaces in a terminal window including form-based input.

### startOfLine

Move the cursor to the start of the current line.

```
startOfLine;
```

### previousLine

Move the cursor to the start of the previous line.

### showCursor

Shows or hides the cursor.

```
showCursor(show: true);
```

### column

Move the cursor to the given column on the current line.

### columns

Returns the number of columns currently displayed by the terminal.

This value can change at any time if the user resizes the terminal window.

### cursorUp

Move the cursor up one row

### cursorDown

Move the cursor down one row

### cursorLeft

Move the cursor to the left one column

### cursorRight

Move the cursor to the right one column

### home

Sets the cursor to the top left-hand corner (column = 0, row = 0)

### row

Move the cursor to the given row.

### rows

Returns the number of rows currently displayed by the terminal.

This value can change at any time if the user resizes the terminal window.

###


# Managing Files And Directories

## Manage files and directories

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

DCli provides a complete set of tools for manipulating files and directories.

DCli also includes the [paths](https://pub.dev/packages/path) package that provides tools for manipulating file paths.

A fundamental task of most CLI applications is the management of files and directories. DCli provides a large set of tools for file management.

### pwd

The getter 'pwd' returns the present working directory.

```dart
print(pwd);
> /home/me
```

Whilst you can change your working directory we don't recommend it. Read the section on the [evils of cd](/dcli-api/the-evils-of-cd).

If you think you need to change your working directory check to see if the DCli function takes a 'workingDirectory' argument.

If you need to spawn another CLI application that needs to run in a specific directory use the '[start](/dcli-api/calling-apps#start)' function.

```dart
'ls'.start(workingDirectory: HOME);
```

If you really think you have no alternative (you are probably wrong) the you can use the Dart method Directory.current.

### Find

The find command lets you explore your file system by searching for files that match a glob (wildcard).

```dart
List<String> results = find('[a-z]*.jpg').toList();
```

The find command starts from the present working directory (pwd) searching for any files that start with the lowercase letters a-z and ending with the extension '.jpg'.

The default action of find is to do a recursive search.

```dart
List<String> results = find('[a-z]*.jpg', workingDirectory: '\' ).toList();
```

If you need to do a search starting from a location other than you current directory you can use the 'workingDirectory' argument which controls where the find starts searching from.

```dart
List<String> results = find('[a-z]*.jpg', workingDirectory: '\', hidden: true ).toList();
```

The find command will ignore hidden files (those starting with a '.') and directories. If you need to scan hidden files then pass 'hidden: true'. If you need to return directories as well as files then use the 'types' argument.

```dart
var progress = Progress((file) => print(file));
find('*.jpg', root: '\'
  , types:[Find.directory, Find.file]
  , progress: progress);
```

If you are process a large amount of results you may want to process them as you go rather than waiting for the full result list to be available.

By passing a 'Progress' into 'find' your progress will be called each time a matching file is found allowing to display the progressive results to the user.

### fileList

Returns the list of files and directories in the current working directory. Use the 'find' function to get a list of any other directory.

```dart
List<String> entities = fileList;
```

### copy

The copy function copies a single file to a to a directory or a new file.

```dart
copy("/tmp/fred.text", "/tmp", overwrite=true);
copy("/tmp/fred.text", "/tmp/fred2.text", overwrite=true);
```

The first example will copy the file 'fred.text' to the '/tmp' directory, the second file also copies the file to the '/tmp' directory but renames the file as it goes.

If the 'overwrite' is not passed or is set to false (the default) an attempt to copy over an existing file will cause a 'CopyException' to be thrown.

### copyTree

The copyTree function allows you to copy an entire tree or selected files from the tree to another location.

The copyTree function takes an optional 'filter' argument which allows you to selectively copy files. Only those files that match the filter are copied.

```dart
copyTree("/tmp", "/tmp/new_dir", overwrite:true, includeHidden:true
   , filter: (file) => extension(file) == 'dart');
```

The above copyTree only copies files from '/tmp' that have an '.dart' extension.

### move

The move function copies a single file to a directory or a file. If the 'to' argument is a file then the file is renamed.

The move function tries to use the native OS 'rename' function however if the destination is on a different device the rename will fail. In this case the move function performs a copy then delete.

```dart
move('/tmp/fred.txt', '/tmp/folder/tom.txt');
```

### moveTree

The moveTree function allows you to move an entire tree or selected files from the tree to another location.

The moveTree function takes an optional 'filter' argument which allows you to selectively move files. Only those files that match the filter are moved.

```dart
moveTree("/tmp/", "/tmp/new_dir", overwrite: true
   , filter: (entity) {
   var include = extension(entity) == 'dart';
   if (include) {
     print('moving: $file');
   }
  return include;
);
```

Like the move function the moveTree attempts an OS level rename but if that fails it resorts to performing a copy followed by a delete.

### delete

The delete function deletes a file.

```dart
delete("/tmp/test.fred", ask: true);
```

If you pass the 'ask' argument to the delete function then the user will be prompted to confirm the delete action.

### deleteDir

The deleteDir function deletes a directory.

```dart
deleteDir("/tmp/testing";
```

If the directory isn't empty then a DeleteDirException will be thrown.

You can delete an entire directory tree using the recursive option:

```dart
deleteDir("/tmp/testing", recursive=true);
```

### createDir

The createDir function creates a directory. If the directory already exists then a CreateDirException will be thrown.

```dart
if (!exists('/tmp/fred/tools')) {
    createDir("/tmp/fred/tools");
}
```

If the parent path doesn't exists then a CreateDirException will be thrown, to avoid this pass the recursive argument

```dart
createDir("/tmp/fred/tools", recursive: true);
```

### touch

The touch function updates the last modified date/time stamp of the passed file. If the 'create' argument is passed and the file doesn't exists then the file will be created. If the file doesn't exists and 'create: true' isn't passed then a 'TouchException' will be thrown.

```dart
touch('fred.txt, create: true');
```

### exists

The 'exists' function checks if a file, directory or symlink exists.

```dart
if (exists("/fred.txt"))
```

### isWritable

Test if a file or directory is writable.

```dart
if (isWritable('/fred.txt'))
```

### isReadable

Test if a file or directory is readable.

```dart
if (isReadable('/fred.txt'))
```

### isExecutable

Test if a file or directory is executable.

```dart
if (isExecutable('/fred.txt'))
```

### isFile

Test if the given path is a file.

```dart
if (isFile('/fred.txt'))
```

### isLink

Test if the given path is a symbolic link.

```dart
if (isLink('/fred.txt'))
```

### isDirectory

Test if the given path is a directory.

```dart
if (isDirectory('/fred.txt'))
```

### setModified

Sets the last modified date/time stamp on give path.. This is similar to touch exception that you can choose the date/time.

```dart
setLastModifed('/fred.txt', DateTime.now());
```

### lastModified

Returns a DateTime reflecting the last modified date/time stamp of the given path.

```dart
DateTime modified = lastModifed('/fred.txt');
```

### calculateHash

Calculates the sha256 hash of a file's content generating essentially a unique signature or checksum for the file.

This is likely to be an expensive operation if the file is large. You can use this method to check if a file has changes since the last time you took the file's hash.

```dart
var digest = calculateHash('/fred.txt');
```


# Environment variables

## Environment Variables

DCli provides tools to manage the environment variables within your DCli script and any child process you call from a DCli script.

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

When a DCli script starts, it loads the set of environment variables from its parent process (usually your shell). The full set of environment variables are available via the `envs` function which returns a map containing key/value pairs for all environment variables.

To access an environment variable called 'COLORTERM':

```dart
var colorTermValue = env['COLORTERM'];
```

You can also set an environment variable:

```dart
env['DART_SDK'] = 'somepath';
```

Once you create or modify an environment variable, then any calls to `env[]` will return the modified value.

If you run a child process via any of the DCli methods then the child process will be passed all of current environment variable.

{% hint style="warning" %}
You CANNOT change the parent shell's environment variables. This is a security restriction imposed by the OS.
{% endhint %}

DCli also exposes a number of commonly used environment variables as global getters.

* HOME - your home directory
* PATH - a list of all the paths that make up your PATH.
* pwd - the present working directory.

```
// home will contain the path to your HOME directory.
var home = HOME;

/// paths will contain a list of the paths contain in your OS PATH environment variable.
List<String> paths = PATH;

paths.forEach((path) => print(path));

print('Your working directory is $pwd);
```

### envs -> Map\<String, String>

Returns a map of all the environment variables inherited from the parent as well as any changes made by calls to \`env\[]=\`.

### PATH

DCli provides a list of methods allow you to modify the PATH. Like any environment variables modifying the PATH will only affect child process you call and not the parent shell.

Methods to manipulate the path include:

* appendToPATH
* prependToPATH
* removeFromPATH
* isOnPATH
* delimiterForPATH

### withEnvironment

The `withEnvironment` function allows you to modify environment variables within the scope of a call.

This can be used to configure a set of environment variables when you run a process that has specific requirements or simply nested code that uses environment variables.

It is particularly useful when writing unit tests as you can set alternate environment variables for each unit test.

```dart
 withEnvironment(() {
     /// run some command with an latered HOME environment variable
     /// any functions called from directly or indirectly from here
     /// will see this scoped environment.
    }, environment: {'HOME': testDir});
```

The `environment` argument is merged with the existing `env` map. Any environment variables passed in via `environment` will replace existing keys in `env`.

No other code will see the modified `env`.


# Calling apps

## Calling other applications

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

The DCli API can run any console (CLI) application.

DCli provides a extensive number of methods to run CLI applications.

DCLI is also being able to process the output of any application it runs.

The importance of this ability is clearly reflected in the no. of ways that the DCli API gives you to run other apps.

### nothrow

DCli has a philosophy of explicit directives. By this we mean; if something doesn't work as explicitly stated then we throw an exception.

For example if you try to delete an directory that doesn't exist then DCli will throw an exception.

```dart
delete('non existant file');
```

When running CLI apps the convention is that an app returns '0' to indicate success.

You can do this in your own DCli scripts via a call to exit

```dart
import 'dart:io';
void main()
{
    exit(0);
}
```

One of the key consequences of this principle is that if you run an app from DCli and that application returns an non-zero exit code then DCli will throw an exception.

In most cases this is the correct action to take.

However some application return a non-zero exit code to indicate something other than a failure. In these cases you need to suppress the exception. A number of methods include a 'nothrow' option will will suppress the normal exception in the case of a non-zero exit code.

Using the 'nothrow' option allows you to obtain the exit code as well as any output from the application.

You also need to use the 'nothrow' option if you need to process any output that went to stderr when a non-zero exit code is returned.

### Treating Strings as commands

DCli extends the String class to provide a simple mechanism for running other CLI applications.

The aim of this somewhat unorthodox approach is to deliver the elegance that Bash achieves when calling CLI applications.

The following example shows how we have added a `run` method to the String class. The `run` method treats the String as a command line that is to be executed.

In this example we run the command 'wc' (word count) on the file 'fred.txt'. The output from the call to 'wc' will be displayed on the console.

```dart
 'wc fred.text'.run;
```

DCli adds a number of methods and operator overloads to the String class.

These include:

* run
* start
* forEach
* toList
* toParagraph
* firstLine
* lastLine
* \| operator

This is the resulting syntax:

```dart
    // run wc (word count) on a file
    // all wc output goes directly to the console
    'wc fred.text'.run;

     // Run echo as a detached process
    'echo into the void'.start(detached: true);

    // run grep, printing out each line but suppressing stderr
    'grep import *.dart'.forEach((line) => print(line)) ;

    // run tail printing out stdout and stderr
    'tail fred.txt'.forEach((line) => print(line)
        , stderr: (line) => print(line)) ;
    
    // run the 'ls' command in the /tmp directory
    'ls'.start(workingDirectory: '/tmp');
```

If you need to pass an argument to your application that contains spaces then use quotes: e.g.

```dart
   'wc "fred nurk.text"'.run
```

Dcli will strip the quotes and pass 'fred nurk.text' as a single argument.

### run

The run command is the simplest option for running an external application.

In runs the application, outputs both stderr and stdout to the console and waits for the application to complete.

```dart
'wc "fred nurk.text"'.run
```

### toList

This is probably one of the most common methods used as it captures any output from the called application and returns it as a list.

```dart
var results = 'wc "fred nurk.text"'.toList(skipLines: 1)
```

### start

Use the start function when you need more control over how the application executes.

#### includeParentEnvironment

By default child processes inherit the parent environment. Set `includeParentEnvironment: false` to prevent that and only pass variables explicitly set via `env['...']` (or none if you cleared them).

#### workingDirectory

One of the most commonly use options is the 'workingDirectory'.

```dart
var results = 'wc "fred nurk.text"'.start(workingDirectory: '/home/me');
```

If you have read the section on the evils of CD then you will understand the need for the 'workingDirectory'. When you pass a workingDirectory to the 'start' command it executes the command ('wc') in the given workingDirectory rather than the user's present working directory (pwd).

#### privileged

If you need to run a command with escalated privileged then set the \[privileged] argument to true.

On Linux this equates to using the sudo command. The advantage of using the 'privileged' option it is cross platform and it will first check if you are already running in a privileged environment.

This is extremely useful if you are running in the likes of a Docker container that doesn't implement sudo but in which you are already running as root.

On Windows setting the priviliged argument to true will cause an exception to be thrown unless you are running as an Administrator.

Calling the 'isPrivileged' function returns true if you are running under sudo/root on posix systems and true if you are running as an Administrator on Windows.

### which

While the 'which' function doesn't run an executable it can be invaluable as it searches your PATH for the location of an executable.

To run an executable with any of the DCli methods you DON'T need to know its location (provided it's on the path) but sometimes you want to know if an executable is installed before you try to run it.

```dart
if (which('grep').found) print('grep is installed');
if (which('grep').notfound) print('grep is not installed');
```

To get the path to the 'grep' command:

```dart
var grepPath = which('grep').path;
```

The 'which' function may find multiple copies of grep on your path in which case it will return each of them in an array in the order that they were found on the path.

In the above example we use the 'path' function to return the first path found for the 'grep' command.

To see all the locations of grep use:

```dart
List<String> where = which('grep').paths
```

You can also use the 'which' function to determine if a particular program is installed:

```dart
if (which('grep').isEmpty) print('grep not installed');
```

Of course in reality we are just seeing if grep is on the path. In theory it could be installed by not on the path.

**Cross Platform which**

The `which` offers built in cross platform support.

On posix systems (Linux, Mac OS) executables normally do not have a file extension. On Windows executables will have a file extension such as '.exe'.

So on posix we have`grep` whilst on Windows we have `grep.exe`.

Windows provides the list of executable extensions in the PATHEXT environment variable.

The `which` funciton uses PATHEXT when searching for matching commands. So if you call:

```dart
which('grep')
```

On a Posix systems we might see:

```dart
which('grep').path == '/usr/bin/grep';
```

On Windows we might see one of:

```dart
which('grep').path == 'C:\Windows\grep.exe';
which('grep').path == 'C:\Windows\grep.bat';
```

If you pass an extension to the which command then DCli will not search for alternate extensions:

```dart
which('grep.exe').path == 'C:\Windows\grep.exe';
```

You can stop which searching for alternate extension by passing `extensionsSearch: false`

```dart
which('grep', extensionSearch: false).notfound == true
```

## Escaping

Prior to DCli 1.10, DCli did not support escaping of command arguments.

DCli provides a number of methods to call an external process. Commands such as `start` and `run` allow you to pass a full command line.

One common problem when passing a full command line is escaping.

Traditionally the backslash character '\\' has been used to escape special characters however DCli aims to be cross platform and this causes problems when running under Windows as the backslash '\\' character is used as a path separator.

Dart also use the the backslash '\\' character to escape which further confuses issues.

To avoid these issues DCli uses the '^' character for command line escaping.

As with all escaping schemes to insert a '^' escape it with a double hat '^^'.

In bash you might write something like:

```bash
cat hello\ world.txt
```

To run the above command using DCli you would write

```dart
'cat hello^ world.txt'.run;
```

Of course a better alternative is to avoid escaping whenever possible. The above command could be written as:

```dart
'cat "hello world.txt"'.run
```

The intent of this command is (imho) much clearer.

## Quotes

DCli aims to replicate bash processing rules for command lines that contain quotes.

Windows however causes some problems.

If you are using the `start` command and passed command arguments that are quoted then AND you set `runInShell` to true then DCli will spawn your command via the Windows Command shell using the `/C` switch.

The `/C` switch does some rather unhelpful processing of quotes:

From the command help:

```
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:

    1.  If all of the following conditions are met, then quote characters
        on the command line are preserved:

        - no /S switch
        - exactly two quote characters
        - no special characters between the two quote characters,
          where special is one of: &<>()@^|
        - there are one or more whitespace characters between the
          two quote characters
        - the string between the two quote characters is the name
          of an executable file.

    2.  Otherwise, old behavior is to see if the first character is
        a quote character and if so, strip the leading character and
        remove the last quote character on the command line, preserving
        any text after the last quote character.
```


# Redirecting output

This page talks about redirecting output from a process (app) that you run using one of the DCLI commands such as 'start'.

If you are not familiar with concepts such as stdout and stdin then have a read of our primer on [stdin/stdout/stderr](/dart-basics/stdin-stdout-stderr).

If you have used bash then you may be familiar with the bash redirect operator '>'. DCli also allows you to redirect output and the most common method we use is a 'Progress'.

So let's have a look at how we use Progress to redirect the output of the 'start' command.

By default, the `start` command prints all output (stdout and stderr) to the console. But what if we want to redirect stdout to a log file?

Passing a Progress to the 'start' command allows you to redirect both stdout and stderr independently.

### redirect stdout to a log

```dart
import 'package:dcli/dcli.dart';

void main() {


import 'package:dcli/dcli.dart';

void main() {

  const pathToLog = 'log.txt';
  print('running ls');
  'ls *'.start(progress: Progress(pathToLog.append));

  print('Displaying the log file');
  cat(pathToLog);
}

```

### Redirect stderr

Redirect stderr to a log whilst still printing to stdout to the console

```dart
void main() {
  const pathToLog = 'log.txt';
  print('running ls');
  'ls *'.start(progress: Progress( print, stderr: (line) => pathToLog.append));

  print('Displaying the log file');
  cat(pathToLog);
}
```

### long hand

The above two examples use tear-offs which make it a little hard to understand what is going on so let's do it the long way:

```dart
void main3() {
  const pathToLog = 'log.txt';
  print('running ls');
  'ls *'.start(
      progress: Progress((line) {
      // the first positional argument to Progress is a lambda which is
      /// called each time a line is written to stdout
    print(line);
  }, stderr: (line) {
     /// the second named argument to Progress is a lambda which is
     /// called each time a line is written to stderr
    pathToLog.append(line);
  }));

  print('Displaying the log file');
  cat(pathToLog);
}

```

## dealing with errors

When a console app writes to stderr it may also exit with a non-zero exit code.

By default, the DCli 'start' command will throw an exception if an app exits with any value but zero.

If you are looking to process the output from stderr in order to take some action when an error occurs, then you need to suppress DCli's default behaviour of throwing an exception.

In this case, you need to pass the 'nothrow' argument to start.

```dart

void main() {
  final errors = <String>[];

  final result = 'ls /fred'.start(
      /// stop the start command from throwing if 'ls' returns a non-zero exit code
      nothrow: true,
      progress: Progress((line) {
        // do nothing, so stdout is suppressed
      }, stderr: (line) {
        // add errors to the [errors] list
        errors.add(line);
      }));

  /// non-zero exit code means we have a problem.
  if (result.exitCode != 0) {
    if (errors[0].contains('No such file')) {
      printerr("The path passed to `ls` doesn't exist");
    }
  }
  
}
```


# Command Line Arguments

A CLI app is only so useful, unless you can pass arguments to your app.

{% hint style="info" %}
Use `dcli create --template=full myproject` to create an example cli app that demonstrates best practices when parsing arguments.
{% endhint %}

Like many languages Dart allows you to pass arguments to your main method..

```dart
void main(List<String> args)
{
    print('Found ${arg.length} arguments.');

    print('The arguments are:');

    for (int i = 0; i < args.lenght; i++) {
        print('arg[$i]=${args[i]}');
    }
}
```

If you DCli script is called test.dart:

```
dart test.dart one two three
> Found 3 arguments.
> The arguments are:
> arg[0] = one
> arg[1] = two
> arg[2] = three
```

You can also stop your app and return an exit code using the exit method.

```dart
import 'dart:io';

void main(List<String> args)
{
    print('Found ${arg.length} arguments.');

    /// stops the progam so no further lines will be executed.
    /// The progam outputs an exit code of 1.
    exit(1);

    print('The arguments are:');

    for (int i = 0; i < args.lenght; i++) {
        print('arg[$i]=${args[i]}');
    }
}
```

## ArgParser

For simple command argument processing you can process the args argument yourself.

If you want to do more complex argument processing then its better to get some help.

The Dart team has very kindly put together the [args](https://pub.dev/packages/args) package which provides advanced argument parsing. The DCli API includes the args package so you do NOT need to add it to your pubspec.yaml dependencies.

You can read all about using the [args](https://pub.dev/packages/args) package on [pub.dev](https://pub.dev/packages/args) but here is a little example of what you can do:

```dart
import 'dart:io';
import 'package:dcli/dcli.dart';

/// This is a full implementation of the linux cli 'which' app.
/// The which command searches the PATH for the passed exe.
void main(List<String> args) {

  /// create the parser and add a --verbose option
  var parser = ArgParser();
  parser..addFlag('verbose', abbr: 'v', defaultsTo: false, negatable: false);

  /// parse the passed in command line arguments.
  var results = parser.parse(args);

  /// get the value of the passed in verbose flag.
  var verbose = results['verbose'] as bool;

  /// The 'rest' of the results are any additional arguments
  /// we only expect one which is the name of the exe we are looking for.
  if (results.rest.length != 1) {
    print(red('You must pass the name of the executable to search for.'));
    print(green('Usage:'));
    print(green('   which ${parser.usage}<exe>'));
    exit(1);
  }

  /// name of the command we will search for.
  var command = results.rest[0];
  var home = env['HOME'];

  List<String> paths = Env().path;

  for (var path in paths) {
    if (path.startsWith('~')) {
      path = path.replaceAll('~', home);
    }
    if (verbose) {
      print('Searching: ${canonicalize(path)}');
    }
    if (exists(join(path, command))) {
      print(red('Found at: ${canonicalize(join(path, command))}'));
    }
  }
}
```

To use the above application:

```dart
dart which.dart ls
Found at: /usr/bin/ls
Found at: /bin/ls
```


# Paths

Generating file paths can be a tedious task and error prone if you want your paths to be cross platform.

Fortunately there is a solution at hand.

The good folk a Google have created the [paths](https://pub.dev/packages/path) package for our use.

The paths package includes a number of global functions that let us create and manipulate file paths.

* join
* dirname
* extension
* basename
* basenameWithoutExtension
* truepath - this is actually a DCli provided method.
* canonicalize

The path package is included as part of DCli so you don't need to add a dependency to your pubspec.yaml.

For a full list of the available functions please refer to the path [API](https://pub.dev/documentation/path/latest/).

## Join

The join function is probably the most used path function. It combines components of a path into a single path.

The best way to understand it is via some examples.

```dart
var tmp = join(rootPath, 'tmp', 'abc', 'image.txt');
print(tmp);
> /tmp/abc/image.txt
```

'rootPath' is a DCli property that is the root directory for your OS. On Linux and Mac OS this is '/', on Windows this is 'C:' (dependent on your current working directory).

```dart
var apps = join(HOME, 'apps');
var tmp = join(apps, 'which.dart');
print(tmp);
> /home/me/which.dart
```

{% hint style="info" %}
HOME is a DCli property which returns the value of the environment variable 'HOME' (in this example /home/me)
{% endhint %}

In the above example we can see that the join function combines two path fragments to form a complete path.

## Dirname

The dirname function returns the directory path of a file path:

```dart
var tmp = join(rootPath, 'tmp', 'abc', 'image.txt');
print(tmp);
> /tmp/abc/image.txt
print(dirname(tmp));
> /tmp/abc
```

The dirname function will also strip the last directory of a path is the passed path doesn't have a filename.

```dart
var tmp = join(rootPath, 'tmp', 'abc');
print(tmp);
> /tmp/abc
print(dirname(tmp));
> /tmp
```

The dirname function is often used to traverse up a directory tree.

```dart
var tmp = join(rootPath, 'tmp', 'abc');
while (tmp != rootPath) {
    print(tmp);
    tmp = dirname(tmp);
}
print(tmp);

> /tmp/abc
> /tmp
> /
```

## Extension

The extension function returns the extension of a file.

```dart
var tmp = join(rootPath, 'tmp', 'abc', 'image.txt');
print(tmp);
> /tmp/abc/image.txt
print(extension(tmp));
> txt
```

## Basename

The basename function returns the filename.

```dart
var tmp = join(rootPath, 'tmp', 'abc', 'image.txt');
print(tmp);
> /tmp/abc/image.txt
print(basename(tmp));
> image.txt
```

basenameWithoutExtension

The basenameWithoutExtension function turns the filename without the extension.

```dart
var tmp = join(rootPath, 'tmp', 'abc', 'image.txt');
print(tmp);
> /tmp/abc/image.txt
print(basenameWithoutExtension(tmp));
> image
```

## truepath

Truepath is s DCli function rather than a 'path' function.

The truepath combines a number of 'path' functions to return an absolute path that has been normalised.

{% hint style="info" %}
A normalised path is one which has had the '..' components resolved.
{% endhint %}

A common mistake made by many CLI applications when reporting errors is to report relative paths.

```dart
/// if you pwd is /usr/home
truepath('adirectory') == '/usr/home/adirectory';
truepath('..\adirectory') == '/usr/adirectory';
truepath('..', 'adirectory') == '/usr/adirectory';
truepath(rootPath, 'usr', 'home', 'adirectory') == '/usr/home/adirectory';

// on windows
truepath(rootPath, 'usr', 'home', 'adirectory') == 'C:\usr\home\adirectory';
```

For the user of your application it can often be difficult to determine what the relative path is relative to. All DCli errors that contain a path call truepath so that the path is absolute and normalised, this gives your users the best chance of correctly identifying the file or path that caused the problem.

You should also normalised any path that a user enters. Using '..' to bypass security checks is a common hacking trick.

## **canonicalize**

If you need to compare to paths you need to both canonicalize your path. As Windows paths are case-insensitive the canonicalize operations returns a all lowercase version of the path which ensures to equivalent paths will return true when a string comparison is performed. The call to canonicalize will also normalize the path.

The only safe way to compare two paths it to compare two absolute paths that have been canonicalized:

```dart
canonicalize(absolute('adirectory')) == '/current/dir/adirectory';
```

The `absolute` call assumes that `adirectory` is in the current working directory. To get the absolute path of a directory relative to some other directory use `relative`.

```dart
canonicalize(relative('adirectory', from: '/home/me')) == '/home/me/adirectory';
```


# Glob Expansion

Glob Expansion refers to the expansion of wildcards (\*.txt) into a list of files.

When you type a command such as 'ls \*.txt' the shell expands the '\*.txt' wildcard (referred to as a glob) to a list of files that match the given wildcard.

As a result of Glob expansion the 'ls' command receives a list of files that end in \*.txt rather than the original glob.

If no files match the glob then the 'ls' command receives the actual glob '\*.txt'.

{% hint style="warning" %}
Glob Expansion does not occur on Windows as neither PowerShell nor Command do glob expansion.
{% endhint %}

Glob expansion is sometimes undesirable. A classic example of this is the linux find command:

```bash
find / -name "*.txt"
```

If the '\*.txt' glob was expanded then the find command would be passed a list of files in the current directory which is clearly not the intent. By wrapping the glob with quotes you are telling bash not to expand the glob but to pass '\*.txt' directly to the find command so it can process the glob against each directory it visits.

DCli uses the same process. If you encase a glob in quotes then DCli will not expand the glob.

```dart
'find / -name "*.dart"'.run;
```

In the above example DCli sees that the glob is wrapped in quotes and as such passes the glob to find without first expanding it.


# Piping

## Piping

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

Now let's pipe the output of one cli command to another.

```dart
('grep import *.dart' | 'head -n 5').forEach((line) => print(line)) ;
```

The above command launches 'grep' and 'head' to find all import lines in any Dart file and then trim the list (via head) to the first five lines and finally print those lines.

Note: when you use pipe you MUST surround the pipe commands with parentheses () due to a precedence issue. In the above example note the parentheses just before the .forEach and the matching one at the start of the line.

What we have now is the power of Bash and the elegance of Dart.


# Locking

DCli provides a NamedLock class which enables you to control access to a resource.

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

There are many scenarios where you only want a single process to access a file or some other resource.

NamedLocks are a co-operative locking mechanism. This means that if some process chooses to ignore the lock then we can do nothing about that.

However if you are running multiple copies of a cli application that you built with the DCli api then you can use a NamedLock to ensure that the apps co-operate with each other.

The NamedLock class tries to be clever and is able to detect if a crashed application has left an old lock lying around. If NamedLock detects this it will release the lock held by the crashed application.

```dart
 NamedLock(name: 'rebuild').withLock(() {
          /// this body will only be called when the lock is taken
          // Do a database rebulid....
        });
```

There a many uses case for a NamedLock, internally we run parallel deployments which require dcli scripts to be pre-compiled. Rather than having multiple deployment tools all trying to compile the tools at the same time we wrap the compile step in a named lock.


# Fetch

The fetch command allows you to send and receive data to or from a server.

Fetch supports http and https but where possible you should always use https.

In its simplest form fetch is often used to download a file from a web server however you can also post data to a web server.

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

DCli allows you to fetch a single web resource with progress information or to simultaneously fetch multiple resources.

## Fetch a single resource

The resource 'sample.aac' will be downloaded and saved to the temporary file 'sample.aac'.

```dart
withTempFile((sampleAac) {
  try {
        String baseURl =
'https://raw.githubusercontent.com/noojee/dcli/master/test/src/functions/fetch_downloads';
        fetch(url: '$baseURl/sample.aac', saveToPath: sampleAac);
    } on FetchException catch (e) {
      print('Exception Thrown: ${e.errorCode} ${e.message}');
    }
    /// print the returned data including any errors.
    if (exists(tmp)) {
      print(read(tmp).toParagraph());
    }
}, create: false
, suffix: 'acc');
```

## Fetch as single resource and show progress

```dart
  withTempFile((sampleAac) {
   try {
       fetch(url: '$baseURl/sample.aac',
            saveToPath: sampleAac,
            onProgress: (progress) {
            print(progress);
       });
    } on FetchException catch (e) {
      print('Exception Thrown: ${e.errorCode} ${e.message}');
    }
    /// print the returned data including any errors.
    if (exists(tmp)) {
      print(read(tmp).toParagraph());
    }
 }, create: false
 , suffix: 'acc');
```

## Fetch multiple resource and show progress

```dart
void get() {
  var sampleAac = fs.tempFile();
  var sampleWav = fs.tempFile();

  fetchMultiple(urls: [
          FetchUrl(url: '$baseURl/sample.aac', saveToPath: sampleAac, onProgress: showProgress),
          FetchUrl(url: '$baseURl/sample.wav', saveToPath: sampleWav)
        ]);
}

void showProgress(FetchProgress progress) {
  print(progress);
}
```

## Post data to a server

Send the data contained in the 'content' variable to httpbin.org.

```dart
 withTempFile((file) {
     try
     {
        const content = 'Hellow World';
        fetch(
            url: 'https://httpbin.org/post',
            method: FetchMethod.post,
            data: FetchData.fromString(content),
            saveToPath: file);
        /// process the json response.
        final map =
            Parser(read(file).toList()).jsonDecode() as Map<String, dynamic>;
        expect(map['data'] as String, equals(content));
        expect(
            (map['headers'] as Map<String, dynamic>)['Content-Type'] as String,
            equals('text/plain'));
    } on FetchException catch (e) {
      print('Exception Thrown: ${e.errorCode} ${e.message}');
    }
    /// print the returned data including any errors.
    if (exists(file)) {
      print(read(file).toParagraph());
    }
  }, create: false);
      
```

## FetchData

When posting data to a server you can provide the data from a number of different sources using the appropriate FetchData constructor.

### FetchData.fromString

Provides data to the fetch method contained in a String.

By default the mimeType is set to text/plain but you can override this by explicitly passing a mimeType

```dart
FetchData.fromString('Hello World', mimeType: 'plain.csv');
```

### FetchData.fromFile

The fromFile constructor uses a file as the source of the data to be posted.

By default fromFile will use the filename's extension to determine the mime type.

You can override the default behaviour by passing the mimeType to the constructor

```dart
FetchData.fromFile('mountains.png');

FetchData.fromFile('mountains', mimeType: 'image/png');
```

### FetchData.fromBytes

The fromBytes constructor allows you to set the source of data as a byte array.

```dart
  withTempFile((pathToData) {
        withTempFile((file) {
          const bytes = <int>[0, 1, 2, 3, 4, 5];

          fetch(
              url: 'https://httpbin.org/post',
              method: FetchMethod.post,
              data: FetchData.fromBytes(bytes),
              saveToPath: file);
        
        }, create: false);
      });
```

### FetchData.fromStream

The fromStream constructor allows you use a stream as the source of the post data.

By default the mimeType is set to 'application/octet-stream' but you can override this by passing an explicit mimeType to FetchData.fromStream.

```dart
withTempFile((pathToData) { 
    withTempFile((file) { 
        const content = 'Hellow World2'; 
        pathToData.write(content);  
            
      fetch(
          url: 'https://httpbin.org/post',
          method: FetchMethod.post,
          data: FetchData.fromStream(File(pathToData).openRead()),
          saveToPath: file);
          
    }, create: false);
  });
```

### Custom Headers

The fetch function allows you to set custom HTTP headers.

The 'Content-Type' header will be overridden by the mimeType in FetchData.

```dart
   withTempFile((file) {
        fetch(url: 'https://httpbin.org/get',
            headers: {'X-Test-Header1': 'Value1', 'X-Test-Header2': 'Value2'},
            saveToPath: file);
            
      }, create: false);
```


# The evils of CD

## CD/pushd/popd are evil

{% hint style="info" %}
For complete API documentation refer to: [pub.dev](https://pub.dev/documentation/dcli/latest/dcli/dcli-library.html)
{% endhint %}

The cd, pushd and popd commands of Bash seem like fun but they are actually harbingers of evil.

I know that they are used everywhere and they seem such an elegant solution but in a script they just shouldn't be used.

So if you shouldn't use cd, pushd or popd what should you do instead?

There a three basic techniques you will use:

* absolute paths
* use the 'start()' method with a working directory
* relative paths

DCli automatically injects the rather excellent package ['path'](https://pub.dev/packages/path) which includes an array of global functions that allow you to build and manipulate file paths to create relative and absolute paths.

You should prefer absolute paths over relative paths.

Such as:

```dart
String filePath = join(HOME, 'directory', 'file.txt');

String dartPath = join('/', 'usr', 'lib', 'bin', 'dart');

// absolute path to your current working directory.
String current = absolute('.');

// create a safe path by replacing the segments (..) with the real path.
String safe = canonicalize(join('..', '..', 'hacker'));

String dirname = dirname(join('usr', 'lib', 'fred.text'));
assert(dirname == '/usr/lib');
```

Often when running an application you need to set the working directory to run the command in.

The following examples runs the command 'git status' from the working directory

/home/yourhome/dev/myproject.

```dart
// run a command using a specific working directory
'git status'.start(workingDirectory: join(HOME, 'dev', 'myproject'));

`
```

With the `path` package at your disposal there is really no need to use cd, pushd or popd.

### Why is cd dangerous?

There are several reasons.

1\) Dart is multi-threaded

> This probably won't be an issue for you as DCli will NEVER start an Isolate and most scripts don't need to use Isolates, but best pratices says that you should assume that one day you might just need to use one, so read on...
>
> Dart and consequently DCli allow you to run multiple threads of execution via Isolates.
>
> The problem is that all of these Isolates running in your Dart process share a single common working directory (CWD or PWD).
>
> This means that if you use CD in one isolate, then all other isolates have their working directory changed under their feet.
>
> Imagine if you are about to do a recusive delete in one isolate and some other Isolate changes the working directory to `/`.
>
> Oops you just deleted your entire file system.

2\) A function forgets to pop

> What happens if you call a function that happens to change the working directory?
>
> Again you can end up deleting your entire file system if the function changes to `/`.
>
> 3\) Another process deletes your working directory What happens if another process deletes your working directory just as you are about to delete all of its contents? If you are using the 'paths' package then it will climb the path until it finds a directory that exists and set that as you new working directory. Your new working directory could well be the root directory.

The correct answer is simply don't use CD/PUSH/POP.

Use relative or preferably absolute paths.


# Assets/Resources

Flutter allows you to bundle assets (graphics, sounds, config files...) along with your flutter application however dart cli applications do not have this ability.

DCli provides a set of asset management tools and an api to work around this limitation.

DCli refers to assets as 'resources' to differentiate them from flutter assets.

{% hint style="info" %}
If you use resources in a package that will be publish to pub.dev, remember there is a 10MB limit on the entire dart package.
{% endhint %}

DCli does this by packaging a resource into a .dart library and then providing an api that allows you to unpack the resources at runtime.

The `dcli pack` command, base64 encodes each file and writes them as a multi-line string into a dart library under src/dcli/resource. The name of the dart library is randomly generated.

The pack command also creates a register of the packed libraries in src/dcli/resource/generated/resource\_registry.g.dart.

DCli expects all resources to located within your dart project under:

```
<project root>/resource
```

You can also pack external resources by creating a [pack.yaml](#external-resources) file.

## Packing resources

To pack your resources run:

```
dcli pack
```

The 'pack' command will scan the '\<project root>/resource' directory and all subdirectories. Each file that it finds will be converted to a dart library under:

```
<project root>/lib/src/dcli/resource/generated
```

Each library name is generated using a md5 hash prefixed with the letter 'A' to make a valid class name. The library name is of the form A\<md5hash>.g.dart

So the following resources:

```
<project root>/resource
                    /images/photo.png
                    /data/zips/installer.zip
```

Will result in to:

```
<project root>/lib/src/dcli/resource/generated
                            /resource_registry.g.dart
                            /A21302b1b380201578fc8ce748f5d9ac8.g.dart
                            /A49bc9b7e40a7f3042a5bbb3e476b4dc4.g.dart
```

The contents of each resource is base64 encoded into a multi-line string. So the photo.png.dart file will something look like:

````dart
class A21302b1b380201578fc8ce748f5d9ac8 extends PackedResource {
  /// PackedResource - local_batman.yaml
  const A21302b1b380201578fc8ce748f5d9ac8();

  /// A hash of the resource (pre packed) calculated by
  /// [calculateHash].
  /// This hash can be used to check if the resource needs to
  /// be updated on the target system.
  /// Use :
  /// ```dart
  ///   calculateHash(pathToResource).hexEncode() == packResource.checksum
  /// ```
  /// to compare the checksum of the local file with
  /// this checksum
  @override
  String get checksum =>
      '14189a469cf7f78af8cd8d4e03815ea72412cea6cbe94779a8db9f736e147300';

  /// <package>/resource relative path to the original resource.
  @override
  String get originalPath => 'lphoto.png';

  @override
  String get content => '''
bG9nUGF0aDogL3Zhci9sb2cvYmF0bWFuLmxvZwoKZW1haWxfc2VydmVyX2hvc3Q6IGxvY2FsaG9zdApl
bWFpbF9zZXJ2ZXJfcG9ydDogMjUKZW1haW
````

## Resource Registry

As part of the packing process DCli also creates a registry of the packed resources. This is done by creating a dart library called:

`<project root>/lib/src/dcli/resource/generated/resource_registry.g.dart`

Each of the packed resources is listed in the register as a map with the 'mount point' as the key.

The `mount point` is the path of the packed resource relative to the `<project root>/resource` directory.

For external resources, you specify a mount point to the project's resource directory that must not collide with any actual resource names under the \<project root>/resource directory.

The contents of the 'resource\_registry.dart' are of the form.

````dart
// ignore: prefer_relative_imports
import 'package:dcli/dcli.dart';
import 'Bbbcdcbeeff.g.dart';
import 'Bfbbcabcfec.g.dart';

/// GENERATED -- GENERATED
///
/// DO NOT MODIFIY
///
/// This script is generated via [Resource.pack()].
///
/// GENERATED - GENERATED

class ResourceRegistry {
  /// Map of the packed files.
  /// Use the path of a packed file (relative to the resource directory)
  /// to access the packed resource and then call [PackedResource].unpack()
  /// to unpack the file.
  /// ```dart
  /// ResourceRegistry.resources['batman.yaml']
  ///     .unpack(join(HOME, '.mysettings', 'batman.yaml'));
  /// ```
  static const resources = <String, PackedResource>{
    'local_batman.yaml': A21302b1b380201578fc8ce748f5d9ac8(),
    'docker_batman.yaml': A49bc9b7e40a7f3042a5bbb3e476b4dc4(),
  };
}
````

## Unpacking resources

DCli provides an API that allows your script to unpack its resources at run time.

```
ResourceRegistry().resources['<relative filename>'].unpack(String localPath)
```

The `resources` field is a map and the key to the map is the original path to the packed file 'relative' to the resources directory.

e.g.

```
  const filename = 'PXL_20211104_224740653.jpg';
  
  final jpegResource = ResourceRegistry.resources[filename];
  
  final pathToConfigs = join(HOME, '.myapp');
  if (!exists(pathToConfigs)
  {
    createDir(pathToConfigs);
  }
  jpegResource!.unpack(join(pathToConfigs, filename);
```

To unpack the resources on the target system use the `ResourceRegistry` class.

The `ResourceRegistory.resources` field is a map of the packed resources. The key is the path of the original resource file relative to the `resource` directory.

Use the `.unpack` method to unpack your resource to a local path on the target system.

```dart
ResourceRegistry.resources['rules.yaml']
    .unpack(join(HOME, '.mysettings', 'rules.yaml'));
```

The values in the resources map is a `PackedResources`. The `PackedResource` includes a `checksum` field. The checksum can be used to see if the expanded resource is the same as the packed resource. You can use this to determine if you need to upgrade the unpacked resource with the latest packed one.

```dart
if (calculateHash(pathToResource).hexEncode() != packResource.checksum)
{
    /// unpack the latest version of the resource.
}
```

## Unpack all resources

You can unpack all resources by interating over the resource values:

````dart
```dart
import 'package:path/path.dart';

 final localTargetDir = join('some', 'local', 'folder');
 for (final resource in ResourceRegistry.resources.values) {
    final localPathTo = join(localTargetDir, resource.originalPath);
    createDir(dirname(localPathTo), recursive: true);
    resource.unpack(localPathTo);
  }

```
````

## External Resources

You can also pack resources that are external to your project by creating a pack.yaml file under your project's tool/dcli directory.

The pack.yaml file allows you to specify a number of files and/or directories.

```yaml
externals:
  - external:
    path: ../template/basic
    mount: template/basic
  - external:
    path: ../template/cmd_args
    mount: template/cmd_args
  - external:
    path: ../template/find
    mount: template/find
  - external:
    path: ../template/hello_world
    mount: template/hello_world
```

### path

The path is an absolute path or path relative to your dart project's root directory.

This is normally used for resources that live outside the projects directory structure but can also specify a file/directory that lives within the project but outside the 'resource' folder.

The path can be a file or a directory.

If path is a directory then the directory is included recursively.

### exclude

When the above `path` key specifies a directory, you may want to selectively excludes some files under the specified directory.

To exclude paths under the `path` key add an exclude section:

```yaml
externals:
  - external:
    mount: template
    path: ../template
    exclude: 
      - project/full/settings.yaml
      - project/full/pubspec_overrides.yaml
```

The list of excluded paths may be a path relative to the root of the `path` directory or an absolute path.

An excluded path may be a file or a directory.

### mount

When a file is packed an entry is added to the resource registry.

To unpack a file you need a key to the file in the registry.

The mount is the key into the resource registry.

For files under the \<project root>/resource directory the mount is their relative path to the resource directory.

So for:

\<project root>/resource/myfile.dart

The mount is `myfile.dart`.

For files and directories specified in the pack.yaml you must specify a mount .

The mount is a virtual path and can be any path you wish provided that it does NOT collide with any other mount specified in pack.yaml or used by any file physically under the \<project root>/resource directory.

**A mount is always a relative path.**

To unpack external resources you use the mount as the key into the ResourceRegistry.

## Ignored files

By default the pack command ignores any hidden files (those beginning with a '.'). To include a hidden file or directory add an explit path statement to that file or directory.

## Limits

If you plan on publishing your project to pub.dev be aware that pub.dev has a maximum package size of 10MB.

The base64 encoding process increases the file size by about 33%.

For apps that you deploy locally the limits are not documented but are probably constrained by your systems memory - so fairly large.

## Automating packing of resources

If you use pub\_release to publish to pub.dev then you can create a pre-release hook to have pub\_release package your resources.


# Cross Platform

Dart and DCli are designed to be cross platform with support for Linux, Mac OS and Windows.

Most of the API's in DCli can be used without consideration for which platform you are running on. There are however a number of issues that you should be aware of.

## Platform

If you need to perform an OS specific operation the you can use the Dart `Platform` class:

```dart
import 'dart:io';
import 'package:dcli/dcli.dart;

void main() {
    if (Plaform.isWindows) {
         // do windows stuff
    }
    else if (Platform.isLinux) {
    /// do some linux stuff
    } else if (Platform.isMacOS)
    {
    // do mac stuff.
}
```

For the most part you will find little need to differentiate between Linux and Mac OS unless you are spawning an OS application.

## Paths

One of the biggest headaches with building cross platform apps is the differences in paths.

Windows uses a drive designator C: and the backslash character \ whilst Linux and OSX use the forward slash /.

Most of the problems around Paths can be avoided by using the <https://pub.dev/packages/path> package which is included in DCli.

So rather than hard coding a path like:

```dart
var path = '/var/tmp';
```

Use the paths package:

```dart
var path = join(rootPath, 'var', tmp);
```

On Windows `path` will be `C:\var\tmp` assuming that your current drive is the C: drive.

On Linux and Mac OS the `path` will be `/var/tmp`.

On Windows if you want to build a path that operates on the current drive without including the drive in the path then use:

```dart
var path = join(separator, 'var', tmp);
```

This will result in:

Linux/MacOS: \`/var/tmp'

Windows: r'\var\tmp'.

Windows will interpret the above path to apply to whatever drive is current the active drive.

The <https://pub.dev/packages/path> package has a collection of functions for manipulating paths that will handle just about every circumstance you need.

## Escaping

DCli is designed to allow the easy development of cross platform scripts that run on Windows, Linux and MacOS.

This presents some problems when building paths and launching child processes.

On Windows the path separator is '\\' whilst on Linux and MacOS it is '/'.

Historically most systems us the '\\' character as the escape character. This proves problematic when you want to create path on Windows as each path separator would need to be double encoded as '\\\\'.

{% hint style="info" %}
You still use the standard \ character when creating Dart strings. The escaping discussion only applies to how DCli parses command line arguments based to functions such as `start` AFTER the normal Dart string escaping has been applied..
{% endhint %}

Example:

```
'git commit \\aswitch foo^ bar'.run;
```

The above string will go through two transformations:

1\) Dart will see the `\\` and output a single `\`

2\) the DCli run command will be given the output of step 1 and when parsing the command line output `foo bar` as a single argument.

DCli recommends using the `join` command (and associated functions) to build paths. If the '\\' was used as the escape character then every call to join would have to be wrapped in a function to escape the resulting path.

Additionally Dart uses the '\\' character as an escape character this can make building strings even harder as you need to double escape each backslash (of course you should be using the join command and not entering the path separator manually!).

To avoid these problem DCli uses the ^ character to escape command line arguments.

It should be noted that this ONLY affects the DCli commands that take a command line argument such as `start`, `run`, `forEach`, `toList` etc.

```dart
'git commit --message=foo^ bar'.run;
```

In this example we are escaping the space before the word bar.

The command will be parsed into the following arguments:

```dart
['git', 'commit', '--message=foo bar']
```

So the key advantage of using `^` is that when constructing paths with tools like `join` you don't need to do any escaping.

```dart
'git ${join(rootPath, 'git', 'myapp')}'.run;
```

The above command works on Windows and posix systems without requiring any escaping.

## Launching Dart Scripts

On each Platform you can run a dart script directly from the command line.

On Linux/Mac OS

```bash
./hello.dart
```

On Windows

```bash
hello.dart
```

There is however an issue when trying to spawn a dart script from with a DCli script or other shell script.

On Linux/Mac OS you can simply run the script (assuming it has a shebang at the top of the script).

```
'hello.dart'.run;
```

On Windows this method won't work as the Dart file association on Windows will only work if you spawn the command via a shell. The problem with spawning the command from a shell is that Windows doesn't appear to return the return value from the spawned script.

The best way to overcome this situation is create an instance of DartScript and run the script using its run method. This technique is guaranteed to be cross platform.

```dart
DartScript.fromFile('hello.dart'.run();
```

## Environment Variables

Environment variables between Windows and posix systems differ significantly.

DCli attempts to abstract some of these differences away.

### HOME

The 'HOME" environment variable works as expected on all platforms.

### PATH

The PATH environment variables at times need special handling on Windows.

The dcli PATH global getter returns a String list of paths, so for simple operations use this function.

For both Windows and posix systems you can't update the path of the parent process. This means that if you are running a DCli script from with in a shell (bash, command, zsh etc) that you cannot change the path of the that shell. This is a sensible security constraint imposed by all operating systems.

You can however modify the PATH for any child process you launch from your DCli script. To modify the PATH of a child process use one of the DCli Env() methods. This rule also applies for any environment variable. If you change any environment variable with DCli then any child process launched (after that point in time) will also see the updated environment variable.

You can change PATH environment in a persistent and DCli provides a number of helper methods.

Using Shell you can update the PATH environment variable in a persistent manner:

```dart
Shell.current.appendToPATH("/usr/me");
Shell.current.prependToPATH('C:\Users\Me\someapp');
```

NOTE: at this point in time not all implementations of Shell in DCli support these operations and they will return false if the operations isn't supported.

Currently the following Shell are supported:

* bash on linux
* Mac OS (append only)
* Windows - Power and Command shells

#### Windows

When updating the Windows PATH DCli will also send a notification to all top level applications that the PATH has been updated. You will however have to restart your Command or Powershell terminal as neither of these shells respond to the notification.

If you need more fine grained control DCli also provides a number of registry functions to directly modify the registry. There are a number of functions like \`regAppendToPath\` to assist. If you use one of the registry functions that include Path in the name then they will also send a Windows notification to all top level applications. Many application will respond to this notification and update their path. Unfortunately neither Command.exe nor Powershell respond to the notification so in both cases you will need to restart the terminal.

#### Mac OS

Only appending a path to the PATH is supported.

#### Linux

Bash is the only Shell with full support for the Shell path methods.

On Linux and Mac OS things are trickier as each shell has its own method of management the PATH environment.

## Built in OS Applications

The set of application supplied by an OS varies considerably so you need to be careful when spawning an application.

Even between Linux distributions there can be differences in what applications are installed by default.

Before spawning an OS application you should check if it exists and whether it is on the PATH. The which function is the most convenient method to do this.

```dart
if (which('ls').found) {
    'ls *.txt'.run;
} else {
    find('*.txt').forEach((file) => print(file);
}
```

Depending on the complexity of the command it may be easier to simply implement it directly in Dart or find a Dart package on <https://pub.dev> that provides the required functionality.

## Glob expansion

When spawning a command DCli follows the rules of the OS on expanding globs.

```dart
'ls *.txt'.run;
```

In the above example '\*.txt' is a glob (a file pattern). On Linux and MacOS the expectation is that DCli will match the '\*.txt' expanding it into a list of files. The `ls` command is then called with that list of files as command line arguments.

```dart
`ls *.txt'.run;

becomes

'ls fred.txt tom.txt'.run.
```

Windows however does not expect globs to be expanded as such we directly pass the glob to the called application.

If you are calling into a native Windows application then everything will work as expected. However if you have written a Dart script which you are now calling you need to understand that the arguments passed to the Dart script will change dependant on which platform the script is running on.

On Windows you will need to have your Dart script expand the glob.

There is a [glob](https://pub.dev/packages/glob) package on pub.dev that will help you to do this.

## Executable Names

Naming conventions for executables differ between Linux/Mac OS and Windows.

On a Linux system a executable normally doesn't have a file extension. On Windows the file extension is .exe.

Further confusion is caused on Windows as when you enter a command such as 'regedit' on a terminal then Windows will search for regedit with a range of extensions such as .exe., .com, .bat, .msi ...

Windows takes the list of extension from the PATHEX environment variable.

To assist with finding the correct extension the DCli `which` function will search for a matching application with each of the extensions in PATHEX.

```dart
which('pub');
> linux -> pub
> windows -> pub.bat
```

This is intended to make it easier to run a command that may have different extensions on different OSs.

You can disable this search behaviour by setting extensionSearch to false:

```dart
which('pub', extensionSearch: false);
```

You can also have Windows apply the the extension search by using the 'runInShell' option on the start command.

```dart
start('pub', runInShell: true);
```


# Posix

Linux and MacOS both include a posix subsystem. This essentially means that they have a set of APIs and commands that conform to the posix stands.

DCli exports a number of posix specific commands.

To access these commands you need to import DCli's posix library.

```dart
import 'package:dcli/posix.dart';
```

## DCli posix specific functions

### chmod

Sets the permissions on a file on posix systems.

### chown

Sets the owner of a file on posix systems.


# Windows

DCli ships with a number of Windows specific functions and classes.

Under the hood DCli uses the [win32](https://pub.dev/packages/win32) package which we recommend if you need additional Windows specific functionality.

The DCli Windows methods also rely heavily on the win32 package's constants such as `HKEY_CURRENT_USER` so in most circumstances you will need to import win32.

To add win32 to you dependencies.

```
pub add win32
```

To access the Windows specific APIs you need to import the windows barrel file.

```dart
import 'package:dcli/windows.dart';
import 'package:win32/win32.dart';
```

## Windows Registry

The Windows Registry is unique to Windows so if you want to write cross platform scripts then you should avoid using the Registry. However in some circumstances this simply isn't possible

In this case use the `Platform.isWindows` method to determine when to use the registry.

```dart
import 'dart:io';
import 'package:dcli/dcli.dart;

void main() {
    if (Plaform.isWindows) {
         regSetString(HKEY_CURRENT_USER, 'Environment', 'PATH_TEST', 'HI');
    }
    else {
    /// do some posix stuff.
    }
}
```

## Windows specific functions

DCli includes:

### regAppendToPath

Appends \[newPath] to the Windows PATH environment variable.

### regIsOnUserPath

Returns true if the given \[path] is on the user's path.

### regPrependToPath

Prepend \[newPath] to the Windows PATH environment variable.

### regGetUserPath

Gets the User's Path (as opposed to the system path) as a list.

### regSetString

Sets a Windows registry key to a string value of type REG\_SZ.

### regSetNone

Sets a Windows registry valueName with a type REG\_NONE.

### regGetString

Gets a Windows registry value o0f type REG\_SZ \[hkey] is typically HKEY\_CURRENT\_USER or HKEY\_LOCAL\_MACHINE.

### regSetDWORD

Sets a Windows registry key to a string value of type REG\_SZ.

### regGetDWORD

Reads a DWORD from the registry.

### regDeleteKey

Deletes an registry key.

### regDeleteValue

Deletes an registry key.

### regGetExpandString

Retrieves a registry value located at \[hkey]/\[subKey]/\[valueName] that is of type REG\_EXPAND\_SZ.

### regSetExpandString

Sets the \[value] of the \[hkey] located at \[hkey]/\[subKey] in the Windows Registry. The \[value] is set to type REG\_EXPAND\_SZ.

### regKeyExists

Tests if a registry key exists.

### regCreateKey

Creates a registry key.


# Docker

DCli is designed to work with Docker.

DCli has a Docker image you can use directly or use in a Docker 'from' statement.

You can also add DCli to an existing Dockerfile.

[Detecting Docker](/dcli-api/cross-platform/docker/detecting-docker)

[Add DCli to a Docker container](/dcli-api/cross-platform/docker/add-dcli-to-a-docker-container)

[Example DCli App in Docker](/dcli-api/cross-platform/docker/ship-a-dcli-app-in-docker)

##


# Detecting Docker

The DCli api allows you to detect if you are running in a Docker container

## DockerShell

DCli has the ability to detect which shell (bash, powershell, zsh etc) that you are running under.

If you DCli app is used as the Docker ENTRYPOINT then your parent won't be a shell.

In this case calling Shell.current will return a DockerShell:

```dart
DockerShell shell = Shell.current;
```

Attributes of DockerShell.

* shell name is 'docker'
* loggedInUser = 'root'
* isPrivilegedUser will always be true

## Detecting if you are running in a Docker

Using DockerShell to detect if you are in a Docker container is not reliable as in some circumstances you DCli app will be run from within a standard shell (bash etc) within Docker

Instead use:

```dart
if (DockerShell.inDocker)
{
    /// do something docker
}
```

This method looks for the presence of /.dockerenv which Docker guarantee will exist.


# Add DCli to a Docker Container

## Adding DCli to your Dockerfile

You can add DCli to your own Dockerfile. This will allow you to run DCli scripts as part of the Docker deployment process as well as running DCli scripts within the final docker container.

DCli is installed into the root user (as is normal for a Docker container). Installers exist for Linux, Windows and Mac OSX.

Just change the wget path to obtain the correct installer:

Linux path

{% tabs %}
{% tab title="Linux" %}

```
RUN wget wget https://github.com/noojee/dcli/releases/download/latest.windows/dcli_install.exe
# TODO correct this path
ENV PATH="${PATH}":/usr/lib/dart/bin:"${HOME}/.pub-cache/bin":"${HOME}/.dcli/bin"
RUN ./dcli_install.exe
```

{% endtab %}

{% tab title="Windows" %}

```
RUN wget wget https://github.com/noojee/dcli/releases/download/latest.windows/dcli_install.exe
# TODO correct this path
ENV PATH="${PATH}":/usr/lib/dart/bin:"${HOME}/.pub-cache/bin":"${HOME}/.dcli/bin"
RUN ./dcli_install.exe
```

{% endtab %}

{% tab title="MacOS" %}

```
RUN wget wget https://github.com/noojee/dcli/releases/download/latest.osx/dcli_install -O dcli_install
RUN chmod +x dcli_install
# TODO correct this path
ENV PATH="${PATH}":/usr/lib/dart/bin:"${HOME}/.pub-cache/bin":"${HOME}/.dcli/bin"
RUN ./dcli_install
```

{% endtab %}
{% endtabs %}

## Compiling a dart package

Now you have dart and dcli in your container you will want to import a project and compile it.

```
# now lets compile a script.
RUN mkdir -p /build/bin
RUN mkdir -p /build/lib
COPY pubspec.yaml /build
COPY bin /build/bin/
COPY lib /build/lib/
# The --install option adds the compiled script to your path.
dcli compile --install bin/<your script>
```

### Upgrading DCli in your docker image.

After building your docker image you may need to force an upgrade of the DCli version.

You can simply recreate your docker image or to save time you can just up use this one trick (sorry) to force docker to just rebuild the DCli install (and subsequent steps in your docker file).

Add the following line just before the call to wget.

If you want to force an upgrade of DCli just increment the no. and run docker build.

```
ARG PULL_LATEST_DSHELL_INSTALL=1
```

## Using the DCli docker image

A Docker image is available which can be used to create a DCli CLI on your system without polluting your OS.

The docker container presents a CLI with dart and DCli pre-installed so you can experiment with DCli or deploy DCli to system instances.

To use the container:

Create a volume so that your scripts are persistent:

```
docker volume create dcli_scripts
```

Attach to the DCli cli.

```
docker run -v dcli_scripts:/home/scripts --network host -it dclifordart/dcli /bin/bash
bash:/> cd /home/scripts
bash:/home/scripts> dcli create hellow.dart
```

The volume is mounted to `/home/scripts` within your dcli container.

vi is included in the container.

Alternatively you can install and run dcli directly from your cli.

## git based dependencies

dart allows you to include dependencies which are pulled from a git repo.

e.g.

```
dependencies:
  gcloud_lib: 
    git: 
      url: git@bitbucket.org:myrepo/gcloud_lib.git 
      path: gcloud_lib
```

If your git repository is public then you don't need to do anything special.

If your git repository is private then calling `pub get` or attempting a `dcli compile` will fail with an auth error.

If this is your scenario then you may need to make your .ssh keys available to the docker build.

This blog article provide a details on how to achieve this.

<http://blog.oddbit.com/post/2019-02-24-docker-build-learns-about-secr/>

The shorter summary is:

```
  var repo = 'yourdockerrepo';
  var image = 'yourimage';
  var version = '1.0.0';
  setEnv('DOCKER_BUILDKIT', '1');
  'docker build --ssh default -t $repo/$image:$version .'.run;
```

With in your docker file your FIRST line MUST be:

```
# syntax=docker/dockerfile:1.0.0-experimental
...

RUN apt-get update
RUN apt-get install --no-install-recommends -y openssh-client

# do you dcli install stuff here

ENV GIT_REP=github.org
# Give git access to your ssh keays
RUN mkdir -m 700 /root/.ssh; 
RUN touch -m 600 /root/.ssh/known_hosts; 
RUN ssh-keyscan $GIT_REPO > /root/.ssh/known_hosts

RUN --mount=type=ssh  dcli compile bin/cmd_dispatcher.dart -o  /home/build/target/cmd_dispatcher
```

## Alpine

If you are using an Alpine based Docker image then you will need to install gclibc.

WARNING: There appear to be some issues around using alpine. I'm seeing network errors (error 69) running pub get. These generally happen toward the end of the process but occur about 80% of the time. You can reproduce them easily by running `pub cache repair`. My suspicion is that its because we are installing glibc when alpine uses mu libc.

```
ENV GLIBC_VERSION 2.31-r0

# Download and install glibc
RUN apk add --update curl && \
  curl -Lo /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
  curl -Lo glibc.apk "https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk" && \
  curl -Lo glibc-bin.apk "https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-bin-${GLIBC_VERSION}.apk" && \
  apk add glibc-bin.apk glibc.apk && \
  /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc-compat/lib && \
  echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
  apk del curl && \
  rm -rf glibc.apk glibc-bin.apk /var/cache/apk/*

# dcli requires ps command 
RUN apk add --update procps
RUN apk add --update wget

# pub requires bash and tar
RUN apk add bash
RUN apk add tar

RUN wget https://github.com/noojee/dcli/raw/master/bin/linux/dcli_install
RUN chmod +x dcli_install
RUN ./dcli_install
ENV PATH="${PATH}:/usr/bin/dart/bin:/root/.pub-cache/bin"
```


# Example DCli app in Docker

DCli is designed to work with docker and makes for an easy method of developing a Docker based app.

The following is an example Dockerfile showing how to ship a single DCli app in Docker

```docker
FROM google/dart as build

RUN mkdir /src
WORKDIR /src
RUN git clone https://github.com/noojee/batman.git

# remove the git clone and uncomment this lines for local dev.
# COPY batman /src/batman

WORKDIR /src/batman

RUN dart pub get
RUN dart compile exe /src/batman/bin/batman.dart -o /batman


# Build minimal  image from AOT-compiled `/batman`
FROM build
COPY --from=build /batman /batman
WORKDIR /
RUN /batman install

# Run a base line and schedule scans.
ENTRYPOINT ["/batman", "--quiet", "--no-colour", "cron", "--baseline", "30 22 * * * *"]

# remove the ENTRYPOINT and uncomment this line to enable interactive debugging.
# CMD ["bash"]
```

The above example use the `batman` project to show the steps required to run a DCli app in docker.

{% embed url="<https://github.com/noojee/batman>" %}

`batman` is a real app and a useful reference.

Of particular note `batman` includes its own cron daemon which allows it to schedule itself without requiring the Docker image to contain cron (which is rather difficult to do).

To build the docker image run:

```
docker build -t <imagename> .
```

`To run the docker image:`

```docker
docker run <imagename>
```

To debug the image, comment out the 'ENTRYPOINT' and uncomment 'CMD'

You can now connect to the docker image:

```docker
docker run -it <imagename> /bin/bash
```

### Publish your docker image

The following DCli script is from the dcli\_scripts project and automates pushing your app into docker hub.

You will need a docker hub account.

Place the script in your dart project tool directory (or alternatively activate dcli\_scripts).

```dart
#! /bin/env dcli

// ignore_for_file: file_names
import 'dart:io';

import 'package:dcli/dcli.dart';

void main(List<String> args) {
  var parser = ArgParser()
    ..addOption('repo',
        abbr: 'r',
        mandatory: true,
        help: 'The name of the docker repository to publish to.');
  var project = DartProject.fromPath('.', search: true);
  var projectRootPath = project.pathToProjectRoot;
  print('projectRoot $projectRootPath');

  ArgResults parsed;
  try {
    parsed = parser.parse(args);
  } on FormatException catch (e) {
    printerr(red('Invalid CLI argument: ${e.message}'));
    exit(1);
  }

  var repo = parsed['repo'] as String;
  var projectName = project.pubSpec.name;
  var version = project.pubSpec.version;
  var name = '$repo/$projectName';

  var imageTag = '$name:$version';
  print('Pushing Docker image $imageTag.');

  print('docker path: ${findDockerFilePath()}');
  print(green('Building $projectName docker image'));
  'docker build -t$imageTag .'.start(workingDirectory: findDockerFilePath());
  print(green('Pushing docker image: $imageTag and latest'));
  var latestTag = '$name:latest';
  'docker image tag $imageTag $latestTag'.run;
  'docker push $imageTag'.run;
  'docker push $latestTag'.run;
}

String findDockerFilePath() {
  var current = pwd;
  while (current != rootPath) {
    if (exists(join(current, 'Dockerfile'))) {
      return current;
    }
    current = dirname(current);
  }
  return '.';
}
```


# Elevated Privileges

Often you need to run a script with elevated privileges.

On Linux and OSX this means using sudo, on Windows it means using 'Run as Administrator'.

DCli abstracts Linux/OSX sudo and Windows Administrator into a single concept of 'elevated privileges'.

In DCli you can check if your script is running with elevated privileges by calling isPrivilegedUser

```
 if (!Shell.current.isPrivilegedUser) {
    printerr(
       'Please restart ${Script.current.exeName} using with elevated privileges');
    exit(1);
  }
```

## Windows

Under Windows elevated privileges are pretty simple.

If any part of your script needs to run with elevated privileges then just use the 'Run as Administrator' option in windows.

You should add a call to `Shell.current.isPrivilegedUser` at the start of the script and force users to restart with the required privileges.

## Linux/OSX

Under Linux/OSX privileged operations for more problematic.

If you need the entire script to run escalated then use the above `isPrivilegedUser` method however often you will want to only use escalated privileges for some of the script which is possible with the `Shell.current.withPriviliged` method.

Read the page on [sudo](/elevated-privileges/sudo) for additional details and some of the problems you will encounter and how to solve them.


# Sudo

You will often want to run a DCli script using sudo.

Using sudo can complicate things:

There are two core issues:

1\) on debian (and probably other distros) sudo has its own path which is unlikely to include the dart or dcli paths.

2\) when trying to run a dart script you may cause the pub cache to be update at which point it will be owned by root and your normally user account won't be able to access it.

If you do this by mistake you can run the following command to fix the problem.

```bash
sudo chmod -r $USER:$USER ${HOME}/.pub-cache
```

3\) You only want parts of your script to run as privileged.

{% hint style="info" %}
The following dart code also works on Windows. The privilege options simply check that you are running as an Administrator and throw an exception if you are not. Running as a Windows Administrator does not have the same problems that sudo introduces.
{% endhint %}

## Solutions

The following provides guidelines on how to solve the problems.

1\) where ever possible avoid using sudo. Of course this often just isn't practical

2\) use the 'start' function and pass in the privileged flag:

```
'chmod +x script.dart'.start(privileged: true);
```

The privileged flag will check if you are running as sudo, if not it will run the chmod script under sudo. This will cause sudo to prompt the user for their sudo password.

This is a good technique as it limits the use of sudo to just those parts of the script that actually need sudo.

3\) compile the script.

Compiling your dart script has a number of benefits.

A compiled script is a completely self contained executable which means it will never cause your pub cache to be accessed which means sudo won't screw it up.

It also fixes the path issues as you don't need dart or dcli on your path for the script to run.

```
dcli compile script.dart
sudo ./script
```

3\) pass your path down to sudo

```
sudo env "PATH=$PATH" <my dcli script>
```

This technique passes your existing user path into sudo which means it can find both dart and dcli.

This method is still dangerous as if dart decides your script needs to be updated then your pub cache will become owned by root.

4\) Use withPrivileges

The intent is to allow you to start your script with sudo but only the parts of your script to that need sudo will actually use it.

We still recommend you compile your script to avoid dart changing permissions on pub-cache.

You do this by starting your script with sudo but immediately downgrading your sudo access when the script starts and then using the withPrivileges method for those parts of your script that need to run as sudo.

```
// get_keys.dart
void main()
{
    /// downgrade script to not run as sudo
    Shell.current.releasePrivilege();
    
    ... do some non-sudo things
    
    /// any code within the following code block will be run
    /// with sudo privileges.
    Shell.current.withPrivileges(() {
        copyTree('\etc\keys', '\some\insecure\location');
    });
}
```

To run the above script:

```
dcli compile get_keys.dart
sudo ./get_keys
```

5\) Pass a password into sudo

If you a trying to run sudo without user intervention then you are likely going to have to pass the password to sudo.

A typically scenario might be calling scripts on a remote system over ssh

he safest way to do this is to:

* create a file with 600 as the password (so only you have read/write access)
* write the sudo password into that file
* create and compile a dart script that can output the password
* run sudo -A to run you command and retrieve the password.

Create a script to ask the user for the sudo password

Name the following script something like sudo\_ask.dart

In the real world creation of the password file would happen in another part of your code base.

```dart
void main() {
  var password = ask('sudo password:');
  var pathToPassword  = 'sudo.p';
  touch(pathToPassword, create: true);
  chmod(600, pathToPassword);
  pathToPassword.append(password);
 
}
```

Create a script to be called by sudo when it needs the password:

Call this script sudo\_askpass.dart

```
void main()
{
    var pathToPassword = 'sudo.p';
    var password = pathToPassword.read().first;
    print(password);
    /// clean up the password file unless you need it again 
    /// during this run.
    delete(pathToPassword);
}
```

Create and compile the DCli script that you want to run under sudo.

as well as the sudo\_ask.dart script.

```
dcli compile sudo_askpass.dart
dcli compile worker.dart
```

Run the script under sudo

```
./sudo_ask.dart
SUDO_ASKPASS=sudo_ask && sudo -A worker
```


# Performance

## Performance

DCli is intended to start as fast as Bash and run faster than Bash.

When you first run your new DCli script, DCli has some house keeping to do including running a `pub get` which retrieves and caches any of your scripts dependencies.

The result is that DCli has similar start times to Bash and when running larger scripts is faster than Bash.

If you absolutely need to make your script perform to the max, you will want to use DCli to compile your script.

### Compiling to Native

DCli also allows you to compile your script and any dependencies to a native executable.

```
dcli compile <scriptname.dart>
```

DCli will automatically mark your new exec as executable using `chmod +x`.

Run you natively compiled script to see just how much faster it is now:

{% tabs %}
{% tab title="Linux" %}

```
./scriptname
```

{% endtab %}

{% tab title="OSX" %}

```
./scriptname
```

{% endtab %}

{% tab title="Windows" %}

```
scriptname.exe
```

{% endtab %}
{% endtabs %}

As this the script fully compiled, changes to your local script file will NOT affect it (until you recompile) and when the exe runs it will never need to do a pub get as all dependencies are compiled into the native executable.

Check out the the --install option to install the script into your path.

You can now copy the exe to another machine (that is binary compatible) and run the exe without having to install Dart, DCli or any other dependency.

{% hint style="info" %}
Once compiled your script will run on any binary compatible machine WITHOUT dart or dcli.
{% endhint %}


# Dependency Management

## Dependency Management

Dart has a large collection of built in packages. You can read about the core packages at:

<https://dart.dev/guides/libraries/library-tour>

However, sometimes you need a specialised package.

There are thousands of third party packages that you can use in your DCli scripts which can be found at:

<https://pub.dev/packages>

{% hint style="warning" %}
NOTE: you can't use Flutter or web packages in your DCli scripts.
{% endhint %}

To use an external package you need to add it as a dependency to your script.

Dart's dependency management is done via a pubspec.yaml file.

Each package includes install instructions which is simply a matter of adding a dependency line to your pubspec and running:

`dcli prepare`.


# Pubspec Managment

## Pubspec Management

The `pubspec.yaml` file is Dart's equivalent of a `makefile`, `pom.xml`, `build.gradle` or `package.json`.

You can see additional details on Dart's pubspec here:

<https://dart.dev/tools/pub/pubspec>

### How we locate your pubspec

Its important to understand that DCli follows the same rules as dart does for locating a pubspec.yaml, with a few additions.

By following the same rules as dart does DCli makes it possible for DCli scripts to work seamless with your current development tools.

Dart and DCli will look for a pubspec.yaml in the scripts directory and then check each parent directory up to the root of the file system for pubspec.yaml. The first one that we find will be used.

### Default Pubspec

If you create you script using dcli create then it will create a default pubpsec.yaml for you with the following dependencies:

```yaml
dependencies:
  dcli: ^0.25.0
  args: ^1.0.0
  path: ^1.0.0
```

You can changed the default set of dependencies by editing \~/.dcli/pubspec.yaml.

The default dependencies are:

* dcli
* [path](https://pub.dev/packages/path)
* [args](https://pub.dev/packages/args)

The above packages provide your script with a swiss army collection of tools that we think will make your life easier when writing DCli scripts.

The 'path' package provide tooling for building and manipulating directory paths as strings.

The 'args' package makes it easy to process command line arguments including adding flags and options to your DCli script.


# DCli tools

DCli ships with a no. of option command line tools to help you creating and writing DCli scripts.

If you are just using the DCli library then you can safely ignore the DCli tools.

If you want to use the DCli tools then you must first install them:

```bash
dart pub global activate dcli
dcli install
```

You can see a full list of `dcli` commands and arguments by running:

```
dcli 
dcli help
dcli help <command>
```

The syntax of `dcli` is:

```
dcli [flag, flag...] [command] [flag, flag...] [arguments...]
```

## flags

DCli supports a global verbose flag: `--verbose | -v`

When passed to dcli it will result in additional logging being written to the cli.

```
dcli -v create hello.dart
```


# Use a shebang #!

A Shebang is a special entry on the first line of your script that tells the OS which command interpreter to use to execute your script.

{% hint style="info" %}
Shebangs are currently only supported on Linux and OSx.
{% endhint %}

By adding a Shebang to the start of you Dart script you can directly run a script from the cli.

Without a Shebang:

```bash
dart hello.dart
```

With a Shebang:

```bash
./hello.dart
```

It's a small difference but rather useful particularly if you are calling one script from another.

{% hint style="info" %}
To use a shebang you must have activated the optional DCli command line tools.
{% endhint %}

You do NOT need the DCli tools if you just want to use the DCli API but they are required if you want to use the Shebang feature.

If you want to use the DCli tools you must first activate them.

```bash
dart pub global activate dcli
dcli install
```

So let's look at how hello.dart looks with a shebang added.

{% hint style="info" %}
The Shebang #! must be the very first line!
{% endhint %}

```dart
#! /usr/bin/env dcli

/// import DCli's global functions 
import 'package:dcli/dcli.dart';

void main() {
  print('Hello World');
}
```

On Linux and OSX you must mark the file as executable for the Shebang to work.

Mark the file as executable:

```bash
chmod +x  hello.dart
```

{% hint style="info" %}
if you used the `dcli create <script>` command then DCli will have already set the execute permission on your script and added the shebang!
{% endhint %}

Now run the script from the cli:

```bash
cli> ./hello.dart
Hello world
cli>
```

You're now officially in the land of DCli magic.

Faster you say?

Read the section on [compiling](/#compiling-to-native) your script to make it run even faster.


# DCli Compile

The compile command will compile your DCli script(s) into a native executable and optionally install it into your PATH.

The resulting native application can be copied to any binary compatible OS and run without requiring Dart or DCli to be installed.

Dart compiled applications are also super fast.

Usage: `dcli compile [-nc, -i, -o] [<script path.dart>, <script path.dart>,...]`

Example:

{% tabs %}
{% tab title="Linux" %}

```bash
dcli compile hello_world.dart

./hello_world
```

{% endtab %}

{% tab title="OSx" %}

```
dcli compile hello_world.dart

./hello_world
```

{% endtab %}

{% tab title="Windows" %}

```
dcli compile hello_world.dart

hello_world.exe
```

{% endtab %}
{% endtabs %}

You may specify one or more scripts and DCli will compile each of them.

If you don't specify any scripts then DCli will compile all scripts in the current directory.

If you use the --install option the compiled exe will be added to your path.

{% hint style="info" %}
DCli copies the executable into \~/.dcli/bin which is added to your path when you run dcli install.
{% endhint %}

{% tabs %}
{% tab title="Linux" %}

```bash
dcli compile --install hello_world.dart

hello_world
```

{% endtab %}
{% endtabs %}

## Compile a package

DCli can also compile a globally activated package.

```bash
dart pub global activate critical_test
dcli compile --package critical_test
critical_test
```

The compiled package will be automatically copied into the \~/.dcli/bin directory which is on your PATH.

Compiling a globally activated package has a number of uses:

* faster startup time
* you are able to copy the resulting executable to any binary compatible machine and run it without installing Dart
* If you switch Dart versions then the executable will still run even if the package isn't compatible with the installed Dart version. This can be useful if you need to run an old version of dart but want access to the latest version of a Dart CLI package.

When compiling a package DCli will create an executable for each of the scripts listed in the packages pubspec.yaml `executables` section.

{% hint style="info" %}
Ensure that \~/.dcli/bin is on your PATH and is before \~/.pub-cache or the globally activate version will run rather than you compiled version.
{% endhint %}

## Flags:

### --noprepare | -nc :

stop DCli from running prepare before doing a compile. Use this option if you know that you script's dependencies haven't changed since the last compile resulting in a faster compile.

### --install | -i :

install the compiled script into the \~/.dcli/bin directory which is on your path. -

### --overwrite | -o :

if the target script has already been compiled and installed, you must specify the -o flag to allow DCli to overwrite it.

### --package | -p

compiles a globally activated package and installs it into the !/.dcli/bin directory.


# DCli Clean

DCli clean deletes all build artifacts including:

* pubspec.lock
* .packages
* .dart\_tools
* all compiled exes

You don't normally need to run DCli clean unless you need to get your directory structure into a pristine state (perhaps before running unit tests). Normally \`dcli prepare\` is a more appropriate tool.

```
dcli clean
```

Your scripts are now ready to run.


# DCli Create

The `dcli create` command makes it easier to create new scripts and templates.

The `dcli create` command can create a project or add a script to an existing Dart project.

When creating a script the Dart project must already exist.

When creating a project `dcli create` performs the following actions:

* creates the project directory
* creates \<bin/\<script>.dart>
* creates pubspec.yaml
* creates analysis\_options.yaml
* marks your script as executable
* adds a shebang #! to the start of your script.
* runs `dcli warmup` in the background.

{% hint style="info" %}
dcli create won't create the pubspec.yaml nor analysis\_options.yaml if you create your new script in an existing dart project.
{% endhint %}

### Create a new project

To create a new project from scratch

Usage: `dcli create <project name>`

Example:

```
dcli create snake
Creating project snake using template console-simple.
DCli warmup started in the background.

Created project snake. 

To run your script:
  cd snake
  bin/snake.dart
```

### Add a script

To add a script to an existing Dart project.

{% hint style="info" %}
It is common practice to have multiple scripts in your bin and tool directories.
{% endhint %}

We recommend that you create scripts in the projects bin directory to compile wit Dart standards.

{% hint style="info" %}
You can actually place a script in any directory and it will work but it's better to stick with the Dart standards for project layout.
{% endhint %}

You may also want to create scripts in your `tool` directory.

{% hint style="info" %}
Scripts in your tool directory should be reserved for tooling to help maintain the project and are not part of your set of public scripts.
{% endhint %}

Usage: `dcli create <script.dart>`

Example:

```
cd snake/bin
dcli create my_script.dart
Creating script my_script.dart using template .
DCli prepare started in the background.

Created script my_script.dart in snake/bin.
To run your script:
  ./my_script.dart
```

{% hint style="info" %}
vscode users: edit the project by typing 'code .' on the command line.
{% endhint %}

As the sample script has a Shebang #! added you can execute it directly:

```
./my_script.dart
```

{% hint style="info" %}
If you run you script immediately after creating it, the background 'warmup' may still be running.
{% endhint %}

In which case you may see the message:

```
./test.dart
Waiting for warmup to complete...
Hello World
```

The warmup process is a once off process and only needs to be run again if you change your dependencies.

The first time you run a given DCli script (created with dcli create), DCli needs to resolve any dependencies by running a Dart `pub get` command and doing some other housekeeping.

If you run the same script a second time DCli has already resolved the dependencies and so it can run the script immediately.

## Templates

You can print a list of available templates by running:

```bash
dcli create --list
```

#### Project Templates

DCli creates projects from a set of templates located in `$HOME/.dcli/template/project`

When you create a project you can specify a template:

```bash
dcli create --template=simple snake 
```

If you don't specify a template name then DCli will use `simple` by default.

DCli supports the following templates:

* simple - simple dart project with a single script in bin
* full - include a lib, test and script in bin
* cmd\_args - example parsing command line args
* find - example using the find function.

You can create custom project templates by copying a dart package into `$HOME/.dcli/template/project/custom`

Each template should be in its own directory under `custom`.

If a custom template has the same name as a standard DCli template then the custom template is used. This allows you to override the standard templates that DCli ships with.

The directory name is used as the template name in the `--template` switch.

When DCli creates a project from a template it:

* creates a directory with the provided project name (e.g. snake)
* copies all files from the given template into the new project directory
* updates the name in the pubspec.yaml file to be the project name passed to `dcli create`
* If the template's `bin` directory contains a `main.dart` then that script is renamed to \<project name>.dart
* If the template's bin directory doesn't contain a `main.dart` then the first .dart script it finds will be renamed \<project name>.dart.

#### Script Templates

DCli creates scripts from a set of templates located in `$HOME/.dcli/template/script`

When you create a script you can specify a template:

```bash
dcli create --template=simple snake.dart 
```

If you don't specify a template name then DCli will use `simple` by default.

DCli supports the following templates:

* simple - simple dart script with an empty main
* cmd\_args - example parsing command line args
* find - example using the dcli find function.

When DCli creates a script from a template it will:

* looking in the template directory for a script called `main.dart` and copy it into the current directory.
* rename main.dart to the script name you passed to the `dcli create` command.

### Flags

The dcli create command accepts the following flags:

\-- foreground :

If the foreground flag is passed the dcli warmup process will be ran in the foreground rather than the use background execution.

Now lets create and run our first DCli script.


# DCli Doctor

The doctor command dumps out your system settings to help in diagnosing problems with your dcli install.

When raising a dcli issue on github please include the output from `dcli doctor`.


# DCli Install

## install

The install command MUST be run after you activate dcli to complete the dcli install.

```dart
dart pub global activate dcli
dcli install
```

The dcli install command creates the \~/.dcli directory, expands the dcli templates and adds \~/.dcli/bin to your path.

You will need to restart your terminal after the install completes for your paths to update correctly.


# DCli Run

## run

Runs the given DCli script.

This command is NOT required.

The recommended way to run a DCli script via ./\<scriptname>.dart.

The `dcli run` command is provided for symmetry and the possibility that someone, someday, may try to auto generate calls to dcli and having a consistent command structure can make this easier.

Usage: `dcli run <script path.dart>`

Example:

```
dcli run my_script.dart
```

Which is equivalent to:

```
./my_script.dart
```


# DCli Warmup

## DCli Warmup

DCli warmup essentially does the same as a pub get. It is provided as a convenience function and you can use a dart pub get interchangeably with DCli warmup.

DCli warmup prepares your project so that you can run any of the project scripts.

When a script is run that has been 'warmed' up, it runs in JIT mode and as such has a slower start time when compared to a compiled script.

The advantage of JIT mode is that it makes it easy to iterate over code changes. You can simply edit your script and immediately run the script.

You only need to run warmup again if you make a change to your dependencies.

If you need faster start times then you should consider compiling your scripts using [`dcli compile`](/dcli-tools-1/dcli-compile).

{% hint style="info" %}
If you edit pubspec.yaml in your project then you need to run `dcli warmup` so that DCli sees the changes you have made.
{% endhint %}

If you change your pubspec.yaml you can call \`dcli warmup\` from anywhere in your project's directory structure.

```
dcli warmup
```


# DCli Pack

The dcli pack command allows you to pack resources (images, config files etc) into your cli app.

Whilst flutter allows you to include assets in a dart executable no such feature exists for dart cli apps. The dcli pack command is designed to fill that void.

A resource is just a file that you want to ship with your package.

To pack a resource in you cli app create a 'resource' directory in the root of your project package.

Place each file in the resource directory.

Run `dcli pack`.

You can also pack resources external to your project by creating a tool/dcli/pack.yaml.

For further details on packing and unpacking resources see the [Asset/Resource](/dcli-api/assets) section.


# Upgrade DCli

When a new version of DCli is released you will want to upgrade to the latest version.

Any of your scripts which use DCli will need to have their pubspec.yaml manually updated pubspec.yaml with the new version of dcli.

Once you have updated the version you need to run pub upgrade:

```
dart pub upgrade
```

If you are using the DCli tools then you will need to upgrade the tools:

We run the same process as we did when installing DCli to upgraded it.

```
dart pub global activate dcli
dcli install
```


# Internal Workings

The DCli public API is almost 100% from of Dart Futures and async statements.

This is intentional as Futures provide almost no benefit in cli applications and actually make it harder to write cli apps.

The Dart api has a single function which can only be used on cli applications which is called 'waitFor'.

The 'waitFor' function essentially removes Futures.

DShell relies heavily on the 'waitFor' function to make writing cli apps easy.


# waitForEx

## waitForEx

DCli goes to great lengths to remove the need to use `Futures` and `await;`there are two key tools we use for this.

`waitFor` and `streams`.

`waitFor` is a fairly new Dart function which ONLY works for Dart CLI applications and can be found in the `dart:cli` package.

The DCli API doesn't expose any futures despite using futures extensively in its internal workings.

`waitFor` allows a Dart CLI application to turn what would normally be an async method (returning a future) into a normal synchronous method by effectively 'absorbing' a future. Normally in Dart, as soon as you have one async function, its async all of the way up.

DCli simply wouldn't have been possible without `waitFor.`

`waitFor` does however have a problem. If an exception gets thrown whilst in a `waitFor` call, then the stacktrace generated will be a microtask based stack trace. These stacktraces are useless as they don't show you where the original call came from.

This is why `waitForEx` was born. `waitForEx` is my own little creation that does three things.

1. capture the current stack using StackTraceImpl
2. calls `waitFor` and catches any exceptions
3. If an exception is thrown it patches the stack trace captured in 1 and merges it with the interesting bits of the microtask exception.

The result is that you get a clean stacktrace that points to the exact line that cause the problem and we have a stacktrace that actually shows where it was called from.


# Contributing

## Overview

The process for contributing to DCli is pretty standard for github projects.

Fork this [dcli](https://github.com/bsutton/dcli) github project and clone it to your local system.

Check the list of Issues for any open bugs or enhancements marked as help wanted or if you want to work on something new then raise an issue describing the work.

If you are working on an issue add a comment to the issue so that other people know that its underway. Make note of when you hope to complete the task so we know if the effort has gone stale.

Make your code changes and submit a Pull Request.

## No Futures

You are likely to need to use futures under the hood but the user level API that DCli exposes MUST not expose any futures. Use waitForEx to absorb any futures.

OK, so there are likely to be exceptions to this rule but please discuss you plans and the need for exposing a Future before starting work so you aren't disappointed when it gets rejected (no good idea is likely to be rejected).

## Exceptions and exit code

All exceptions thrown MUST be extended from DCliException.

Invalid command line arguments MUST throw an exception that derives from CommandLineException

Return codes are pure evil and they are the reason Exceptions were invented.

If any OS command is called and it returns a non-zero exit code then you MUST throw an exception derived from DCliException. We do support the 'nothrow' option on a number of commands as non-zero exit codes don't always mean that the call failed.

## Global Settings

The Settings class is the correct place to store any global settings.

## Coding Standards

Note:

* dcli uses the lints package lint rules and as such ALL code MUST be fully compliant. Suppressing lint warnings will normally not be accepted.
* Use of `dynamic` is almost never acceptable.

## Things to check:

* Ensure that your code has been formatted using dart format before committing your code.
* Ensure that you code has no warnings or errors with the exclusion of TODOs.
* Comment your code.
* Fully Document any methods/functions/classes that are exposed as part of the public API
* Ensure that methods/functions are short and readable. Split your methods if they start getting large.
* Include unit tests for your code.
* Ensure that your code doesn't break any existing unit tests.
* Use good function names and variables. Abbreviations are rarely acceptable.
* Be nice to your mother.

## The development cycle

Start by forking the dcli project on git hub

<https://help.github.com/en/github/getting-started-with-github/fork-a-repo>

Now clone the fork to your local machine:

```
cd ~
git clone https://github.com/YOUR-USERNAME/dcli
```

You are now ready to start making a contribution to DCli.

The dcli source code ships with a no. of tools (written in dcli) to help you cycle between developer and user mode.

The dcli tools are located under the dcli/tool directory.

#### activate\_local.dart

Running activate\_local will update your dcli install to use your local source rather than the dcli installed into your pub-cache.

To run activate\_local:

If your dcli development tree is located at \~/dcli

```
cd dcli
tool/activate_local.dart 
```

To switch back to the pub-cache version of dcli

```
pub global activate dcli
```

###

### Raising a Pull Request (PR)

<https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request>

Your PR should be created on a separate branch to allow it to be merged and tested against the DCli repository before we merge it into the DCli main branch.

To have your code changes accepted into the main DCli repository on github you will need to create a PR.

##


# Creating a release

DCli uses the package [pub\_release](https://pub.dev/packages/pub_release#-analysis-tab-) (written in DCli) to create releases.

Start by installing pub\_release.

```
pub global activate pub_release
```

Commit and push all of you code changes.

pub\_release performs the following tasks

* Incrementing the version no.
* runs 'dcli pack' to pack the templates.
* formatting code
* generating release notes
* checking that all code is committed
* pushing the release
* git tagging the release with the new version no.
* publishing the the new release to pub.dev

Run pub\_release

```
cd dcli
pub_release
```

Answer the questions asked.

Job done.


# Running Unit tests

### Running unit tests.

Running DCli unit tests can be a little tricky as they perform write operations on your file system. In particular the 'install' unit tests will delete your dcli installation which is rather inconvenient.

Dcli uses the dart critical test package for running unit tests.

pub global activate critical\_test

Whilst you can run most tests from the vs code unit test framework there are number of tests that require pre-setup as well as sudo/admin privileges. The critical test package has the ability to run pre/post hooks to establish the environment to run these tests.

## Docker

Alternatively you can run the unit tests from a docker container as noted below.

The unit tests are, by design, destructive as they need to delete your current dcli install and recreated it.

To avoid interfering with your local system the DCli source includes a number of docker containers to run your unit tests.

Under the `dcli/docker/test` folder you will find docker containers and dcli scripts to run those containers.

There are two methods available for running the scripts:

* use your local dcli source
* clone from git hub

Use the local source is safe and slightly faster so is the preferred option.

## Run from Local source

`install.local.df and install.local.dart`

The install.local pair allow you to run the unit tests using your local dcli source. To run the install unit tests run:

```
cd dcli/docker/test
./install.local.dart
```

## Run for git clone

`install.clone.df and install.clone.dart`

This pair runs the install scripts by first cloning the git repo and running the scripts against the cloned repo.

The docker file pulls the repo from the main dcli github site. You may want to modify the repo to point to your forked repo.

To run the dcli install unit tests from the cloned git repo run:

```
cd dcli/docker
./install.clone.dart
```


# Implemention support for a shell

DCli provide access to an abstracted interface to a shell's underlying functions through the Shell class.

DCli provides various levels of support for a number of shells including:

* ash
* bash
* cmd
* dash
* fish
* power shell
* sh
* zsh

## Adding support for a shell

Adding basic support for a shell is fairly easy:

1\) copy an existing shell implementation from lib/src/shell

2\) modify the implementation to match your shell's implementation details

3\) register your shell with by adding it to the '*shells' array in lib/src/shell/shell\_detection.dart*

### *Basic Shell support*

To implement the minimal level of shell support you will need to provide implementations for:

* Constructor 'withPid
* name
* hasStartScript
* isCompletionSupported
* hashCode
* operator ==

### Privileged User

On linux we have the concept of sudo and on Windows an Administrator. DCli exposes these concepts as a 'privileged' user.

The Shell.isPrivilegedUser method allows DCli to determine if the current script is running under a privileged user. The libraries posix\_mixin.dart, cmd\_shell.dart and power\_shell.dart all have separate implementations of this. You may be able to use one of these existing implementations but you may need to implement your own.

### Install support

DCli attempts to automate as much of the DCli and dart installation process as possible.

Ideally your shell should support configuring DCli and dart. There are standard implementations for both of these actions that you should normally be able to use.

The platform specific implementation for Windows is in windows\_mixin.dart and for Linux and MacOS in posix mixin.dart.

These mixins should work for any shell so normally your shell should just 'with' the appropriate mixin.

The one exception is support updating the paths for DCli and Dart.

The DCli installer calls Shell.addToPath in order to achieve this.

You will most likely need to implement a custom implementation of Shell.addToPath. Have a look at bashshell.dart and cmd\_shell.dart for example implementations.

### Completion Support

Some shells offer tab completion.

At the time of this writing DCli only supports tab completion for bash.

To implement tab completion for other shells you need to:

Override the lib/src/shell/Shell.dart methods :

* isCompletionSupported
* isCompletionInstalled
* installTabCompletion

You then need to implement a tab completion system for your shell.

For bash DCli ships the executable bin/dcli\_complete.

If your Shell's completion tooling allows/expects the use of an executable to provide the tab completion support you may modify the dcli\_complete.dart library to also provide completion for you shell.

You can use the `Shell.current` method to determine if you shell is being run when the dcli\_complete is executed.


# Templates

Templates are a work in progress.

DCli ships a number of templates which are intended to be used with the dcli create command.

Currently the dcli create uses one fixed template.

Going forward the intent is to add a --template switch to the create command so the user can choose which template to use.

## Creating templates

Each template must live in its own dart project as a subproject in github.

A template should consist of a complete dart project including an analysis\_options.yaml file that conforms to the DCli lint standards (copy dcli's existing file).


# References

## References

Projects I referenced (stole stuff from) when making this package:

<https://pub.dev/packages/dscript_exec>

<https://pub.dev/packages/dartx>

<https://pub.dev/packages/completion>


# Projects

The following is a list of open source projects that use DCli.

These can be a handy jumping off point to see how other people use DCli

| Project           | Description                                                                                            | Repo                                           |
| ----------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------- |
| critical\_test    | Enhanced tooling to run and re-run your dart unit tests allowing you to focus on failed tests.         | <https://github.com/noojee/critical_test>      |
| batman            | System integrity scanner for PCI compliance.                                                           | <https://github.com/noojee/batman>             |
| dcli\_scripts     | Grab bag of handy utilities for maintaining your dev environment.                                      | <https://github.com/noojee/dcli_scripts>       |
| dextract          | Extracts files from multiple archive format.                                                           | <https://github.com/noojee/dextract>           |
| dswitch           | Provides instant switching between dart versions.                                                      | <https://github.com/noojee/dswitch>            |
| dvault            | Utility to encrypted files into a vault using RSA.                                                     | <https://github.com/noojee/dvault>             |
| nginx-le          | Docker container and cli tooling for nginx with built in certbot certificate acquisition and renewals. | <https://github.com/noojee/nginx-le>           |
| OSX-KVM-Installer | Install OSX under kvm on linux.                                                                        | <https://github.com/relf108/OSX-KVM-installer> |
| pub\_release      | Automates releasing projects to pub.dev                                                                | <https://github.com/noojee/pub_release>        |


# Code

The Examples section is intended to be a grab bag of samples that demonstrate various techniques that you can use to solve problems with DCli.

Feel free to use any of the code as your own.

| Example                                        | Description                                                                                                                                                                                                                   |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Hello World](/examples/overview/hello-world.) | A bare bones hello world example.                                                                                                                                                                                             |
| [dcompress](/examples/overview/dcompress)      | <p>Utility that is able to dcompress multiple file types. It uses the file extension to select the appropriate tool to decompress the file with.</p><p>It does require that the appropriate OS tool is already installed.</p> |
| [dpath](/examples/overview/dpath)              | Prints a list of the PATH directories and checks that each one is valid.                                                                                                                                                      |
| [dmysql](/examples/overview/dmysql)            | Allows you to save the connection arguments to the mysql cli command to connect to multiple databases. Once configured you can connect by typing `dmysql <dbname>`                                                            |
| [dshell](/examples/overview/dshell)            | An example REPL shell.                                                                                                                                                                                                        |
| [dwhich](/examples/overview/dwhich)            | Dart implementation of the class `which` command bit it also validates the PATH and prints all occurances of the app that it finds in the order they appear on the PATH.                                                      |
| [dipaddr](/examples/overview/dipaddr)          | Prints out each of the local IP address bound to your machine in a human readable format.                                                                                                                                     |


# hello world.

The classic hello world.

```dart
#! /usr/bin/env dcli

import 'package:dcli/dcli.dart';

void main() {
    print('hello world');
}
```


# dcompress

Decompresses a number of different file formats.

It does require that the appropriate tool is installed.

Supports:

* zip
* tar.gz
* tar
* xz
* rar

```dart
./dcompress.dart file.zip
```

```dart
#! /usr/bin/env dcli
import 'dart:io';

import 'package:dcli/dcli.dart';

/// dcompress <compress file.>
/// de-compresses a variety of file formats.
void main(List<String> args) {
  var parser = ArgParser();
  parser.addFlag('subdir',
      abbr: 'd',
      defaultsTo: false,
      help: 'Extracts the file to a subdirectory');
  var results = parser.parse(args);

  var extensionToCommand = <String, String>{
    '.tar.gz': 'tar -zxvf %F',
    '.tar': 'tar -xvf %F',
    '.xz': 'tar -xvf %F',
    '.rar': 'unrar e %F',
    '.zip': 'unzip %F'
  };

  if (results.rest.length != 1) {
    print('Expands a compressed file.');
    print('');
    printerr(red('You must provide the name of the file to expand.'));
    print('The file will be expanded in the current working directory.');
    exit(1);
  }

  var tarFile = results.rest[0];

  if (!exists(tarFile)) {
    printerr(red("The passed file ${truepath(tarFile)} doesn't exist"));
    exit(2);
  }

  var cmd = extensionToCommand[extension(tarFile)];

  if (cmd != null) {
    cmd = cmd.replaceAll('%F', tarFile);
    try {
      cmd.run;
    } catch (e) {
      if (e is RunException && e.exitCode == 2){
        printerr(red('The extractor for $tarFile $cmd could not be found.'));
      }
      // otherwise supress the exception as the command will print its own error.
    }
  } else {
    printerr(red('The file $tarFile does not have a know extension.'));
    printerr(green('Supported extensions are:'));
    for (var key in extensionToCommand.keys) {
      printerr('  $key');
    }
    exit(1);
  }
}
```


# dpath

Prints and validates that each path PATH exists.

```dart
 ./dpath.dart 
Test:  ✔ /usr/local/sbin
Test:  ✔ /usr/local/bin
Test:  ✔ /usr/sbin
Test:  ✔ /usr/bin
Test:  ✔ /sbin
Test:  ✔ /bin
Test:  ✔ /usr/games
Test:  ✔ /usr/local/games
Test:  ✔ /snap/bin
Test:  ✔ /usr/lib/dart/bin
Test:  ✔ /usr/lib/dart/bin
```

```dart
#! /usr/bin/env dcli

import 'package:dcli/dcli.dart';
import 'package:args/args.dart';

/// dpath appname
/// print the systems PATH variable contents and validates each path.

const String tick = '''\xE2\x9C\x93''';

const String posixTick = '''\u2714''';

const String cross = 'x';

void main(List<String> args) {
  var parser = ArgParser();
  parser..addFlag('verbose', abbr: 'v', defaultsTo: false, negatable: false);

  for (var path in PATH) {
    var pathexists = exists(path);

    if (pathexists == true) {
      print('Test:  $posixTick ${canonicalize(path)}');
    } else {
      print(red('Test: $cross ${canonicalize(path)}'));
    }
  }
}
```


# dmysql

The dmysql.dart script is intended to be a time save if you connect to a mysql cli on a regular basis.

dmysql allows you to save the connection details and then each time you want to connect you just need to pass in the database name:

## To see the command line options:

```
dart dmysql.dart 
You must provide the database name
Connects you to a mysql cli pulling settings (username/password...) from a local settings file.

To connect to a db:
   dmysql <dbname>

To connfigure settings for a db:
  dmysql --config <dbname>
  
-c, --[no-]config    starts dmysql in configuration mode so you can enter the settings for the given db
```

## To configure your database:

```bash
dart dmysql.dart --config mydb
host: [] 127.0.0.1
port: [3306] 
user: [] root
password: [] <root password>
```

## To connect to your db:

```
dart dmysql.dart mydb
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 29
Server version: 10.3.22-MariaDB-1ubuntu1 Ubuntu 20.04

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [mydb]> 
```

## dmysql.dart

```
#! /usr/bin/env dcli

import 'dart:io';
import 'package:dcli/dcli.dart';
import 'package:args/args.dart';
import 'package:settings_yaml/settings_yaml.dart';

/// Connects you to a mysql cli pulling settings (username/password)
/// from a local settings file.
/// Use

var pathToDMysql = '$HOME/.dmysql';

void main(List<String> args) {
  var parser = ArgParser();
  parser.addFlag('config',
      abbr: 'c',
      defaultsTo: false,
      help:
          'starts dmysql in configuration mode so you can enter the settings for the given db');

  var results = parser.parse(args);
  var rest = results.rest;

  if (rest.length != 1) {
    printerr(red('You must provide the database name'));
    showUsage(parser);
  }

  var dbname = rest[0];
  var pathToDbSettings = join(pathToDMysql, dbname);

  if (results['config'] as bool) {
    config(dbname, pathToDbSettings);
  } else {
    if (!exists(pathToDbSettings)) {
      printerr(red('You must first configure your database using --config'));
      showUsage(parser);
    }
    launch(pathToDbSettings);
  }
}

void launch(String pathToDbSettings) {
  var settings = SettingsYaml.load(pathToSettings: pathToDbSettings);
  var host = settings['host'] as String;
  var port = settings['port'] as int;
  var user = settings['user'] as String;
  var password = settings['password'] as String;
  var dbname = settings['dbname'] as String;

  'mysql -h $host --port=$port -u $user --password="$password" $dbname'.run;
}

void config(String dbname, String pathToDbSettings) {
  if (!exists(dirname(pathToDbSettings))) {
    createDir(dirname(pathToDbSettings), recursive: true);
  }
  var settings = SettingsYaml.load(pathToSettings: pathToDbSettings);

  settings['dbname'] = dbname;
  settings['host'] = ask(
    'host:',
    defaultValue: settings['host'] as String,
     validator: Ask.any([
       Ask.fqdn,
       Ask.ipAddress(),
       Ask.inList(['localhost'])
    ]));
  );

  settings['port'] = int.parse(ask('port:',
      defaultValue: (settings['port'] as int ?? 3306).toString(),
      validator: Ask.integer));

  settings['user'] = ask('user:',
      defaultValue: settings['user'] as String, validator: Ask.required);

  settings['password'] = ask('password:',
      defaultValue: settings['password'] as String,
      validator: Ask.required,
      hidden: true);

  settings.save();
}

void showUsage(ArgParser parser) {
  print('''
Connects you to a mysql cli pulling settings (username/password...) from a local settings file.

${green('To connect to a db:')}
   dmysql <dbname>

${green('To connfigure settings for a db:')}
  dmysql --config <dbname>
  ''');
  print(parser.usage);
  exit(1);
}
```

## pubspec.yaml

```
name: dmysql
version: 1.0.0
description: Saves db connection settings and connects to a db cli
environment: 
  sdk: '>=2.6.0 <3.0.0'
dependencies: 
  args: ^1.0.0
  dcli: ^0.24.1
  path: ^1.0.0
  settings_yaml: ^2.0.0

dev_dependencies: 
  pedantic: ^1.0.0
```


# dshell

The dshell example demonstrates building a toy command line shell (just like Bash).

It has three built in commands (ls, cd and exit) and allows you to run any other CLI application that's on your PATH.

To run this script you need to [install dart](/dcli-tools-1/dcli-install)

```yaml
mkdir shell
vi dshell.dart # paste contents of dshell.dart below
vi pubspec.yaml # paste contents of pubspec.yaml below
pub get
dart dshell.dart
```

dshell.dart

```dart
#! /usr/bin/env dcli

import 'dart:io';
import 'package:dcli/dcli.dart';

/// A toy REPL shell to replace your bash command line in just 50 lines of dart.
void main(List<String> args) {
  // Loop, asking for user input and evaluating it
  for (;;) {
    var line = ask('${green(basename(pwd))}${blue('>')}');
    if (line.isNotEmpty) {
      evaluate(line);
    }
  }
}

// Evaluate the users input
void evaluate(String command) {
  var parts = command.split(' ');
  switch (parts[0]) {
    case 'ls':
      ls(parts.sublist(1));
      break;
    case 'cd':
      Directory.current = join(pwd, parts[1]);
      break;
    case 'exit':
      exit(0);
      break;
    default:
      if (which(parts[0]).found) {
        command.start(nothrow: true, progress: Progress.print());
      } else {
        print(red('Unknown command: ${parts[0]}'));
      }
      break;
  }
}

/// our own implementation of the 'ls' command.
void ls(List<String> patterns) {
  if (patterns.isEmpty) {
    find('*', root: pwd, recursive: false, types: [Find.file, Find.directory])
        .forEach((file) => print('  $file'));
  } else {
    for (var pattern in patterns) {
      find(pattern,
              root: pwd, recursive: false, types: [Find.file, Find.directory])
          .forEach((file) => print('  $file'));
    }
  }
}
```

The required pubspec.yaml

pubspec.yaml

```yaml
name: dshell
dependencies: 
  dcli: ^0.32.0
```


# dwhich

Dart implementation of the `which` command.

```dart
#! /usr/bin/env dcli
import 'dart:io';
import 'package:dcli/dcli.dart';

/// dwhich appname - searches for 'appname' on the path
void main(List<String> args) {
  var parser = ArgParser();
  parser.addFlag('verbose', abbr: 'v', defaultsTo: false, negatable: false);

  var results = parser.parse(args);

  var verbose = results['verbose'] as bool;

  if (results.rest.length != 1) {
    print(red('You must pass the name of the executable to search for.'));
    print(green('Usage:'));
    print(green('   which ${parser.usage}<exe>'));
    exit(1);
  }

  var command = results.rest[0];

  for (var path in PATH) {
    if (verbose) {
      print('Searching: ${truepath(path)}');
    }
    if (!exists(path))
    {
	printerr(red('The path $path does not exist.'));
 	continue;
    }
    if (exists(join(path, command))) {
      print(red('Found at: ${truepath(path, command)}'));
    }
  }
}
```


# dipaddr

Prints each of the IP addresses bound to your PC without all of the croft.

```dart
./dipaddr.dart
name: enp39s0
  0) 192.168.1.1
name: virbr0
  0) 192.168.12.1
name: docker0
  0) 172.0.0.1
```

```dart
#! /usr/bin/env dcli

import 'dart:io';

void main() {
	NetworkInterface.list(includeLoopback: false, type: InternetAddressType.any)
    	.then((List<NetworkInterface> interfaces) {
        interfaces.forEach((interface) {
          print('name: ${interface.name}');
          var i = 0;
          interface.addresses.forEach((address) {
            print('  ${i++}) ${address.address}');
          });
        });
    });
}
```


# gnome launcher

## gnome launcher

The following dcli script creates a gnome launcher. You can use this to launch any dcli script (or any app in general) from the gnome menu.

```dart
#! /usr/bin/env dcli

import 'dart:io';
import 'package:dcli/dcli.dart';

String name;
List<String> categories;
bool terminal;
String comment;
String iconPath;
String exePath;

var parser = ArgParser();

///
/// Creates a gnome launcher.
///
void main(List<String> args) {
  Settings().setVerbose(enabled: false);

  parser.addOption('name',
      help: 'Name of the application to be displayed in the gnome menu');

  parser.addMultiOption('categories',
      defaultsTo: ['Development'],
      help:
          'Gnome list of categories. Controls where in the gnome menu the entry appears. e.g. \'Development\'');

  parser.addOption('terminal',
      defaultsTo: 'true',
      help:
          'If true (the default) then the app is launched in a terminal window.');

  parser.addOption('comment', help: 'Adds a comment to the Gnome menu item');

  parser.addOption('iconPath', help: 'Path to an icon for the Gnome menu');
  parser.addOption('exePath', help: 'Path to an executable to be run');

  var parsed = parser.parse(args);

  name = getRequiredString(parsed, 'name');
  comment = parsed['comment'] as String;
  terminal = getBool(parsed, 'terminal');
  categories = getList(parsed, 'categories');
  iconPath = getPath(parsed, 'iconPath');
  exePath = getRequiredPath(parsed, 'exePath');

  writeDesktopEntry();
}

String getRequiredPath(ArgResults parsed, String option) {
  var path = getRequiredString(parsed, option);

  if (!exists(path)) {
    print(red('The provided path for $option does not exists.'));
    showUsage();
  }

  return truepath(path);
}

String getPath(ArgResults parsed, String option) {
  if (!parsed.wasParsed(option)) {
    return null;
  }

  var path = parsed[option] as String;

  if (!exists(path)) {
    print(red('The provided path for $option does not exists.'));
    showUsage();
  }

  return truepath(path);
}

String getRequiredString(ArgResults parsed, String option) {
  if (!parsed.wasParsed(option)) {
    print(red('You must provide the --$option argument'));
    showUsage();
  }
  return parsed[option] as String;
}

bool getRequiredBool(ArgResults parsed, String option) {
  if (!parsed.wasParsed(option)) {
    print(red('You must provide the --$option argument'));
    showUsage();
  }
  return parsed[option] == 'true';
}

bool getBool(ArgResults parsed, String option) {
  return parsed[option] == 'true';
}

List<String> getRequiredList(ArgResults parsed, String option) {
  if (!parsed.wasParsed(option)) {
    print(red('You must provide the --$option argument'));
    showUsage();
  }

  return parsed[option] as List<String>;
}

List<String> getList(ArgResults parsed, String option) {
  return parsed[option] as List<String>;
}

void showUsage() {
  print('');
  print(green('Usage:'));
  print('./add_gnome_laucher.dart [options]');
  print(parser.usage);
  exit(-1);
}

void writeDesktopEntry() {
  var path = join(HOME, '.local', 'share', 'applications',
      '${name.replaceAll(RegExp('[^a-zA-Z0-9_]'), '_')}.desktop');
  // create desktop.ini
  if (exists(path)) {
    delete(path);
  }

  var content = StringBuffer();

  content.write('''
[Desktop Entry]
Version=1.0
Type=Application
Name=$name
''');

  if (comment != null) content.write('Comment=$comment\n');

  content.write('Categories=${categories.join(';')}\n');

  content.write('Terminal=$terminal\n');

  if (iconPath != null) content.write('Icon=$iconPath\n');

  if (terminal) {
    content.write('''gnome-terminal -e 'bash -c "$exePath;bash"' ''');
  } else {
    content.write('Exec=$exePath\n');
  }

  path.write(content.toString());

  print('Created: (in ${truepath(path)})\n');

  cat(path);
}
```


# build CLI apps in dart - part 1

> [Read part 2](https://medium.com/@bsutton.noojee/dshell-build-console-apps-in-dart-part-2-39719cb6d051)

DCli is a library and development environment for building console apps using Dart.

> DCli runs on Linux, MacOS and Windows.

Dart you say, what is this Dart thing?

Dart is a relatively new programming language from Google with a Java/JavaScript heritage with the bad bits taken out. Dart is becoming a mainstream language, if you are interested as to why, google flutter.

> If you are looking to learn Dart, DCli is an easy way to get started.

If you want to learn more about Dart, the Dart language tour is a great place to start:

<https://dart.dev/language>

But I digress, let’s get back to talking about DCli.

DCli was designed to make building CLI scripts and even whole CLI apps a joy.

So DCli starts with Dart and then adds a library of commands that mimic many of the operations of bash. This combination creates the perfect building blocks for a CLI script.

> Before we get started, a quick word about developing with Dart and specifically with DCli.\
> When developing scripts we often don’t have a GUI so an IDE isn’t always available. With that in mind DCli is a pure CLI application so you can easily develop DCli scripts using Vi or whatever other CLI editor you prefer. Having said that, I recommend using an IDE whenever it’s available. A debugger, syntax highlighting, auto complete and automatic insertion of import statements will seriously improve your productivity.\
> My recommendation for an IDE is Visual Code. It’s lightweight, supported by Google and does a nice job of getting the job done.\
> I’ve included a link at the bottom of this article regarding installing Visual Code.

But let’s dive in and look at some examples of DCli.

If you would like you can work through examples as we go by installing DCli.

Start by installing Dart:

<https://dart.dev/get-dart>

Now install DCli:

```
dart pub global activate dcli
dcli install
```

You’re now ready to start developing with DCli.

Let’s have a look at an example DCli script.

hello\_world.dart

```dart
#! /usr/bin/env dcli
import 'package:dcli/dcli.dart';

void main() {
  print('Hello World');
}
```

To run the above script:

run hello\_world.dart

```bash
chmod +x hello_world.dart
./hello_world.dart
```

Note: if you are on Windows the 'chmod' command isn't required.

The first time you run a script, DCli needs to do some housekeeping. DCli (or in reality DArt) downloads any dependencies that your script requires.

Don’t worry about the start time of your script, the second time you run the script the start time will be much faster. If you need serious speed DCli can compile your script to a native executable. But more about that in part 2.

Let’s pull the code apart line by line.

Line 1 is called a ‘shebang’. It essentially tells your OS that the script is to be executed with DCli. You should include this in all your DCli scripts.

Line 2 imports the dcli package making all of its yummy goodness available.

Line 4 is the entry point for the script.

Line 5: we print ‘hello world’

Rather than creating a script yourself let DCli do it for you.

Type: dcli create hellow\.dart

```bash
dcli create hellow.dart
Creating project.
Running pub get...
Resolving dependencies...
+ args 1.5.2
+ charcode 1.1.2
+ collection 1.14.12
+ dcli 0.25
+ equatable 1.0.2
+ file 5.1.0
+ file_utils 0.1.4
+ globbing 0.3.0
+ intl 0.16.1
+ logger 0.8.2
+ matcher 0.12.6
+ meta 1.1.8
+ money2 1.3.0
+ path 1.6.4
+ pub_semver 1.4.2
+ pubspec 0.1.3
+ quiver 2.1.2+1
+ recase 3.0.0
+ source_span 1.5.5
+ stack_trace 1.9.3
+ string_scanner 1.0.5
+ term_glyph 1.1.0
+ uri 0.11.3+1
+ utf 0.9.0+5
+ yaml 2.2.0
Changed 25 dependencies!
Making script executable
Project creation complete.
To run your script:
   ./hellow.dart
```

>

The ‘dcli create’ command creates a sample ‘hello world’ dart script, does all the required housekeeping, and marks the script as executable.

{% hint style="info" %}
dcli ships with a number of starter templates that provide common starting points for writing scripts. Run 'dcli create help' for details.
{% endhint %}

## Calling external app <a href="#id-12db" id="id-12db"></a>

One of bash’s superpowers is that it can call any external application. Well so can DCLI.

```dart
#! /usr/bin/env dcli
import 'package:dcli/dcli.dart';

void main() {
  'grep error /var/log/syslog'.run;
}
```

Line 5 of the above example runs the grep command. Any output from grep is written directly to the console.

{% hint style="warning" %}
For Dart users, this code may look a little confusing. How do you run a String? DCli uses the Dart ‘extensions’ feature to extend the String class. In this case, we have added a ‘run’ property. Watch out for additional String overloads in the examples below.
{% endhint %}

## Using forEach <a href="#id-67dc" id="id-67dc"></a>

We can ‘run’ a command but how do we process the output? Simple, we use the ‘forEach’ method:

```dart
#! /usr/bin/env dcli
import 'package:dcli/dcli.dart';

void main() {
    'grep error /var/log/syslog'.forEach((line) => print(line));

  'grep error /var/log/syslog'.forEach((line) { print('matched $line'); });

  'grep error /var/log/syslog'.forEach((line) => print(line), stderr:(line) => print(red(line)));
}
```

{% hint style="info" %}
Dart supports anonymous functions (or more broadly lambda’s and closures) just as Javascript and Java do. The above forEach function takes a lambda.
{% endhint %}

In line 5, the lambda is the part:

```dart
(line) => print(line)
```

Essentially ‘(line)’ is an argument passed to the lambda. This is essentially the ‘method signature’ of the anonymous function. Each time grep outputs a line, the forEach method calls the above lambda.

The ‘line’ argument will contain the line output by grep to stdout.

The `=>` operator (sometimes referred to as a ‘fat arrow’) tells us that this lambda is expecting a single expression on the right-hand side of the `=>` operator. In this case, we pass the line argument to the print statement which prints the line to the console.

> In case you are not familiar with ‘stdout’ every CLI app has three file handles passed to it by the OS. stdin, stdout, and stderr. Stdin can be read and contains any data piped to the app, the Dart print command writes to stdout (which normally goes to the console) and stderr is where a CLI app should write any error messages to. Read the page on [stdout/stderr/stdin](/dart-basics/stdin-stdout-stderr) for more details.

Let's now compare line 5 to line 7.

Line 5: … .forEach((line) => print(line));

Line 7: … .forEach((line) { print(‘matched $line’); });

What we are seeing here are the two forms of a Dart lambda.

Line 5 uses the fat arrow which expects a single expression.

Line 7: drops the fat arrow in exchange for a statement block ‘{}’.

The advantage of the statement block is that you can include multiple statement (each terminated by a semi-colon).

The second point of interest in line 7 is the use of ‘$line’ in the print statement. This is a Dart string interpolation feature. You can insert any variable into a string by preceding it with a ‘$’. You can in fact include any expression by encapsulating the expression with ${}. e.g. print('${line.substring(3)}').

Line 9 now gets a little more interesting:

```dart
… .forEach(
(line) => print(line)
    ,stderr:(line) => print(red(line)));
```

So what’s going on here? The first half of the line is recognizable from line 5, but what about the second half:

```dart
stderr:(line) => print(red(line))
```

Some of this makes sense:

```dart
(line) => print(red(line))
```

This looks just like the lambda we previously used to print lines to the console. You can probably guess that the call to ‘red(line)’ changes the colour of the line written to the console to red.

{% hint style="info" %}
dcli provides a number of functions for applying colour to text by using the ansi terminal escape sequences. DCLI also supports cursor positioning commands via the Terminal class.
{% endhint %}

But what is the ‘stderr:’ all about?

When you run a command like grep, it outputs any text to stdout, but if an error occurs it writes the error message to stderr. When we used the ‘.run’ method with grep, both stdout, and stderr are written to the console. But when we use forEach, as we did in Lines 5 and 7 we are only writing stdout to the console and essentially suppressing stderr (just like sending it to /dev/null).

Suppressing stderr is not such a good idea but often convenient so forEach lets us ignore stderr. If we want or need to process stderr then that’s where the ‘stderr:’ syntax comes in.

If you are a Dart programmer you will immediately realize that this is a named parameter. But for the non-Dart programmers let’s stop for a moment and explain named parameters.

> You can read more on Dart parameters in the above mentioned Dart language tour or in this blog article I wrote on choosing parameters:
>
> <https://onepub.dev/Blog?id=sppsdavuhn>

Dart functions and methods support three types of arguments.

Positional, optional, and named.

When declaring a named argument in a Dart method we use braces to designate it. So the signature of the forEach method is:

```dart
forEach(LineAction stdout, {LineAction? stderr});
```

> Dart types are 'null be default'. LineAction is a type which can never be null so you can't pass a null to the stdout argument. LineAction? is the same type but it may null so you can pass null to the stderr argument or just don't pass it and it will default to null.

When calling a function with a named argument we use the name and a colon. e.g.

```dart
forEach((line) => print(line), stderr: (line) => print(line));
```

The first positional argument is:

```dart
(line) => print(line)
```

The second named argument is:

```dart
stderr: (line) => print(line)
```

The stderr named argument is optional which is why we could write Line 5 without mentioning stderr.

forEach is probably one of the most important methods in DCLI as we use it repeatedly to process lines of data.

## Let’s talk about piping. <a href="#id-0af9" id="id-0af9"></a>

Another great feature of bash is its ability to call multiple applications and ‘pipe’ the data from one application to the next. Well, we can do the same with DCLI.

```dart
(‘grep error /var/log/syslog’ | ‘head -n 5’ | ‘tail -n 1’.)forEach((line) 
    => print(‘The fifth error is: $line’);
```

The above line calls grep to find all the lines containing ‘error’, passes the results to the ‘head’ command which outputs just the first 5 lines, then tail outputs just the last line of those five, and finally we use Dart to print the fifth line.

> In reality I rarely use piping with DCLI as there are usually better ways with the Dart and DCLI libraries to achieve the same results.

## Built-in commands <a href="#f592" id="f592"></a>

DCli provides a Swiss army knife of built-in functions for building CLI scripts.

> For bash users; Dart supports top level functions just as bash does however unlike bash, the functions can be in any order. Dart also supports classes.

```dart
find('*.png', recursive: false).forEach((line) => print(line);
```

Find is one of the many built-in commands that ships with DCli. By default, find does a recursive search from the current directory but in this case, we only want to search the current directory so we pass the optional named argument ‘recursive’ with a value of false. Once again we see the forEach method in use to process each of the filenames returned by find.

Alternatively, we may want to store the found png files in a list. The DCli commands that support forEach also supports ‘toList’ so if we want to save the list of png files we can simply do:

```dart
var files = find('*.png').toList() ;
```

To create a directory we do:

```dart
createDir('/home/me/a/path/to/far', recursive: true);
```

The optional, named argument ‘recursive’ tells DCli to create any intermediate paths that don’t already exist.

To ask the user a question:

```dart
var answer = ask('Y/N');
```

To delete a file:

```dart
delete('notneeded.txt');
```

To move a file:

```dart
move('from path', 'to path');
```

To check if a file or directory exists:

```dart
if (exists('/home/does/it/exist')) {
    print('found it')
};
```

You can access and set environment variables with the ‘env’ function:

```dart
var username = env['USERNAME'];
env['password'] = 'a password';
```

DCli also directly exposes some environment variables such as:

```dart
HOME — your home directory

PATH — a String array containing the paths on your PATH.
```

## Paths, Paths and more Paths <a href="#id-3c23" id="id-3c23"></a>

When writing CLI scripts you tend to spend a lot of your time manipulating directory paths. Dart makes this easy through neat ‘path’ package.

To use the `path` package we need to add it as a dependency. When you create a DCLI script, DCLI automatically creates a 'pubspec.yaml' file which is used to hold the list of dependencies as well as other configuration details. `pubspec.yaml` is akin to C's make file or Nodes `package.json` file.

To add the `path` dependency to your pubspec.yaml run:

```
dart pub add path
```

To use the `path` package in a script we need to import it at the top of the dart file.

```
import 'package:path/path.dart';
```

The path package provides a set of global functions that allow you to create and manipulate directory paths.

Some of the most commonly used are:

```dart
import 'package:path/path.dart';

join('/home', 'my') == '/home/my'
canonicalize('/home/../home') == '/home'
absolute('test') == '/home/me/test'
basename('/home/fred.txt') == 'fred.txt'
dirname('/home/fred.txt') == '/home'
```

You can see additional details on the path package at:

{% embed url="<https://pub.dev/packages/path>" %}

### Packages, packages and more packages <a href="#id-489f" id="id-489f"></a>

One of DCLI's great strengths is how easy it is to extend DCLI. The Dart eco-system includes thousands of packages that provide all sorts of functions.

{% embed url="<https://pub.dev>" %}

You can find a list of packages specifically designed for writing CLI scripts on the OnePub website:

<https://onepub.dev/search?searchProvider=Category&category=develop.cli>

## Accepting Arguments <a href="#id-90ae" id="id-90ae"></a>

To make your CLI script useful you will more than likely want to process arguments passed to your script. Dart is similar to C and Java in that its entry point is called ‘main’ and it takes an array of arguments.

```dart
#! /usr/bin/env dcli
import 'dart:io';
import 'package:dcli/dcli.dart';

void main(List<String> args) {
    
  print('${args.length} were passed');
  int index = 0;
  for (var arg in args)
  {
    print('arg $index = $arg);
  }
  exit(1);
}
```

Line 2 imports Dart’s io library so we can use the ‘exit’ function below.

Line 5 shows us the main entry point. The main method returns void which means we need to use the ‘exit’ function to return an exit code from the script.

Line 5 also declares that main takes a List of Strings called ‘args’. Dart supports Generics. I’ve provided some references at the bottom of this article on Dart and using Generics with Dart.

Line 7 prints the no. of arguments passed. Again we are using the Dart ‘$’ notation that lets us embed variables into a Dart String.

### Summary <a href="#id-404c" id="id-404c"></a>

There is a lot more to DCli but I think you can see by the above examples that DCli is a simple and expressive method of writing CLI scripts.

You should now have enough information to start writing basic CLI scripts using DCli.

Well, I think that’s enough for one day.

Part 2 will build a complete CLI app using DCli.

{% embed url="<https://github.com/bsutton/dcli>" %}

I’m also looking for collaborators or you could write your own article about DCli :)

Regards,

Brett

### [**Read part 2.**](https://medium.com/@bsutton.noojee/dshell-build-console-apps-in-dart-part-2-39719cb6d051) <a href="#id-7d34" id="id-7d34"></a>

References:

Dart packages: [https://pub.dev](https://pub.dev/packages/dshell)

Note you can NOT use packages designed for Flutter or Web.

DCli: [https://pub.dev/packages/dcli](https://pub.dev/packages/dshell)

DCli git repo: [https://github.com/onepub-dev/dcli](https://github.com/bsutton/dshell)

Path: <https://pub.dev/packages/path>

Args: <https://pub.dev/packages/args>

A great getting-started guide for dart

{% embed url="<https://dart.dev/guides/language/language-tour>" %}

An overview of Dart Generics.

Generics: <https://www.tutorialspoint.com/dart_programming/dart_programming_generics.htm>

Visual Code: links to installing visual code an installing the required extensions.

<https://dartcode.org/>

*


# build CLI apps in dart - part 2

In [part 1](https://medium.com/@bsutton.noojee/dshell-build-console-apps-in-dart-a2d8b76b13be), we took a whirlwind tour of DCli.

In part 2, let’s take some of those pieces and build an app or two.

Before we move forward just a couple of thoughts and some answers to some of the questions I received after part one.

After my initial article the first question was; what magic is DCli doing and how does that magic affect what I can do with Dart?

The answer is very little.

{% hint style="info" %}
DCli allows you to use any Dart feature that is supported in a console app
{% endhint %}

The second question is, why doesn’t DCli use futures?

The answer is that DCli does use futures and you too can use futures in a DCli script.

The longer answer is; that in a CLI script, futures are a pain and for most requirements don’t provide any advantage. DCli goes to considerable lengths to shield you from having to use futures so unless you are doing something tricky; stay away from them.

With that done let’s get back to building our first app.

> You can see some additional sample apps in the [example's](/examples/overview) section of this manual.

Every Linux system ships with the ‘**which**’ command. The ‘which’ command is used to find which directory an application is run from.

For example, if you type:

```
which grep
```

‘Which’ will report:

```
/bin/grep
```

‘Which’ searches each directory in your PATH until it finds the one containing the grep command.

Our version of ‘which’, called **dwhich**, searches for the command and validates each path as it goes. We are also going to include a verbose flag to print progress messages.

Start by creating a DCli script:

```
dcli create dwhich.dart
```

To make life easier we are going to use a couple of helper packages, `args` for parsing command line arguments and `path` for manipulating directory paths. So we first need to add these two packages as dependencies. From the command line (in the `dwhich` directory that dcli just created run:

```
dart pub add args
dart pub add path
```

Now copy the below contents into your dwhich.dart script.

```
#! /usr/bin/env dcli
import 'dart:io';
import 'package:args/args.dart';
import 'package:dcli/dcli.dart';
import 'package:path/path.dart';

/// dwhich appname - searches for 'appname' on the path
void main(List<String> args) {
  var parser = ArgParser();
  parser.addFlag('verbose', abbr: 'v', defaultsTo: false, negatable: false);

  var results = parser.parse(args);

  var verbose = results['verbose'] as bool;

  if (results.rest.length != 1) {
    print(red('You must pass the name of the executable to search for.'));
    print(green('Usage:'));
    print(green('   which ${parser.usage}<exe>'));
    exit(1);
  }

  var command = results.rest[0];

  for (var path in PATH) {
    if (verbose) {
      print('Searching: ${truepath(path)}');
    }
    if (!exists(path))
    {
	printerr(red('The path $path does not exist.'));
 	continue;
    }
    if (exists(join(path, command))) {
      print(red('Found at: ${truepath(path, command)}'));
    }
  }
}
```

Given our requirements we need to pass two arguments to dwhich:

```
dwhich [-v] <appname>
```

The -v flag is optional and \<appname> is the name of the application we are going to search the path for.

To make processing the command line arguments easy we are going to use the ArgParser class from the ‘[args](https://pub.dev/packages/args)’ package that we added as a dependency.

So let’s break things down.

Line 7 we create an instance of the ArgParser

Line 8 we tell the ArgParser that we can accept an optional flag on the command line. The user can either type ‘ —verbose’ or ‘-v’ to cause ‘dwhich’ to output verbose details.

Line 10 parses the command args and gives us the results.

Line 12 extracts the ‘verbose’ flag from the results map and converts it to a bool.

Line 14 checks ‘results.rest’ to see if the expected `appname` argument was passed. The `appname` is stored in ‘results.rest’ which is a simple String list containing all of the arguments passed to main(), after the verbose switch was removed.

Line 22 takes the first argument from ‘result.rest’ which contains the name of the application that the user wants to search for and stores it into the variable ‘command’.

Line 23 uses DCli's ‘PATH’ global variable to loop through all of the paths in the OS’ PATH environment variable. DCli conveniently converts the PATH environment variable to a List\<String> containing each of the paths.

Line 27 validates that each path included in PATH is valid and prints an error if it isn’t. Here we use ‘printerr’ rather than ‘print’. ‘printerr’ writes to stderr whilst ‘print’ writes to stdout.

> You should always use printerr to print error messages.

Line 32 uses the ‘join’ function from the ‘path’ package to create a path by joining the current ‘path’ and the ‘command’ and then tests if the command exists at that path.

Line 33: print a message when we find the command. Note the use of the function ‘truepath’. Truepath is a convenience function provided by DCli. Truepath combines three operations into one. The following two lines give the same result:

```
/// if your current working directory is /home/me/dwhich’;
truepath(‘apps’, bin’, ‘..’, ‘dart); == ‘/home/me/dwhich/apps/dart’
canonicalize(absolute(join(‘apps’, bin’, ‘..’, ‘dart));
```

So why do we need to do all of that?

Line 33 prints the command's path for the user. To make the user's life easier you should always print an absolute path. It is no end of frustration for a user to read an error message that mentions a path but only gives a relative path. Whilst sometimes it will be obvious what path the file is relative to, often it’s not. So good practice is to always print an absolute path.

The canonicalize call is for safety. Hackers have often used THIS ONE TRICK (sorry) to break out of sandboxes.

Where does the following path point to?

```
/home/me/../../usr
```

When canonicalized, the above path reduced to:

```
/usr
```

So by canonicalizing the path we make it easier to read and if your code is checking for a prefix of /home (thinking that’s safe) then the call to canonicalize will show your code that the path isn’t actually safe.

So use truepath whenever you show a user a path or when you need to validate a path.

So we now have a ‘dwhich’ command and it was surprisingly simple to implement.

Let’s try and run it. If you created the script using ‘dcli create’ the script is ready to run:

```
./dwhich.dart grep
```

or

```
./dwhich.dart -v grep
```

If you created the script by hand then you must first mark it as executable (not required on Windows):

```
chmod +x dwhich.dart
./dwhich.dart grep
```

Remember that the first time you run your script, DCli needs to do some housekeeping!

## Make it go faster <a href="#id-6a03" id="id-6a03"></a>

Let’s make our dwhich command go faster by compiling it.

Run

```
dcli compile dwhich.dart
```

This will output an executable called ‘dwhich’.

Let’s run our new executable:

```
./dwhich
```

Notice how much faster the compiled ‘dwhich’ starts.

## Install and copy to other servers <a href="#fb70" id="fb70"></a>

We can also install our dwhich command into the OS PATH by passing the ‘—install’ flag to the compile command.

```
dcli compile — install dwhich.dart
```

DCli will copy the resulting executable to the \~/.dcli/bin directory which was added to your path when you installed DCli.

You can now run dwhich as:

dwhich grep

**But wait there’s more;**

Now that we have compiled our ‘dwhich’ command we have a single file executable.

When DCli compiles a script it includes all of the script's dependencies and a minimal Dart runtime. A compiled DCli script is typically about 8MB in size.

**But wait there’s even more….**

Now we have a compiled executable we can copy just the executable to any binary-compatible machine and run it (linux -> linux, windows->windows). There is no need to install Dart nor DCli on the target machine.

Hopefully, you are now starting to get a feel for how the pieces of DCli fit together.

Let’s build one more app before we finish up.

## duntar.dart <a href="#id-7f15" id="id-7f15"></a>

I don’t know about you, but I often have to untar a file or a .tar.gz file and I can never remember the correct set of switches to get the correct result.

So I built a little script that remembers for me; after all isn’t that what scripting is all about?

```
#! /usr/bin/env dcli
import 'dart:io';
import 'package:args/args.dart';
import 'package:dcli/dcli.dart';

/// duntar <tarfile>
/// untars a file
void main(List<String> args) {
  var parser = ArgParser();
  var results = parser.parse(args);

  if (results.rest.length != 1)
  {
    print('untars a .tar or .tar.gz file');
    print('');
    printerr(red('You must provide the name of the file to untar'));
    print('The file will be untared in the current working directory');
    exit(1);
  }

  var tarFile = results.rest[0];

  if (tarFile.endsWith('.tar.gz'))
  {
    'tar -zxvf $tarFile'.run;
  }
  else if (tarFile.endsWith('.tar'))
  {
    'tar -xvf $tarFile'.run;
  }
  else
  {
    print("The tar file $tarFile does not have a know extension of '.tar.gz' or '.tar'");
  }
}
```

So duntar takes the name or path of a tar file or tar.gz file, checks the extension, and runs untar with the correct switches.

So once again let’s pull the command apart.

Line 9 we have seen before. Declare an ArgParser and ask it to parse the command line arguments.

Line 12 checks that the tar name was passed by checking the size of ‘results.rest’. I should note at this point that we really didn’t need ArgParser as we could have simply accessed ‘args’ directly.

Line 14 and friends print a message telling the user to try again.

Line 21 extracts the name of the tar file from the results.rest array.

Line 23 checks if the tarFile ends with .tar.gz and if so runs the correct tar command.

Line 27 checks again for a .tar file and again runs tar.

Line 31 is a catch all incase they passed in a file that isn’t supported.

## Homework <a href="#id-9949" id="id-9949"></a>

So for some homework why don’t you try to make duntar support some additional file types such as .zip or 7z. You could end up with a single app that will uncompress any file type.

## Wrap up <a href="#ef5e" id="ef5e"></a>

So that’s it for this part.

To me at least DCliis beginning to feel like a really useful tool. I love Dart and it’s really nice being able to script in a modern language.

I’ve dropped a few more examples in a git repo.

[https://github.com/onepub-dev/dcli\_scripts](https://github.com/bsutton/dshell_scripts)

Feel free to issue a pull request and contribute your own little tool. They don’t need to be high quality, just useful.

Finally, if you’re new to Dart, DCli is a great way to get started as it lets you work directly with the language rather than through the prism of a large framework like Flutter.

### Links as promised: <a href="#b886" id="b886"></a>

DCli — [https://pub.dev/packages/dcli](https://pub.dev/packages/dshell)

DCli scripts — [https://github.com/onepub-dev/dcli\_scripts](https://github.com/bsutton/dshell_scripts)

The above example code and more little tools.

ArgParser — <https://pub.dev/packages/args>

Money2 — <https://pub.dev/packages/money2>

A completely unrelated package that I wrote that lets you parse, format, and do maths with Money and Currencies.


# Dealing with permissions

When writing cli apps you will invariably have to deal with file permissions and access rights in general.

DCli provides a number tools and methods to help deal with permissions including tools that assist with cross platform development (linux/windows/osx).

## Run an app with escalated privileges

One of DCli's strengths is its ability to spawn child cli apps:

```
'grep debug *.log'.run
```

The above command will spawn grep as a child process and print its output to the cli.

Under Linux and OSX


# 3rd Party console packages

Here is a list of 3rd party console packages that you might find useful when writing cli apps.

\# Interact

A collection of functions for obtaining user input.

{% embed url="<https://pub.dev/packages/interact>" %}


# Dart on Linux - the perfect CLI tooling

If you haven't already heard, Google and Canonical have released a [joint statement ](https://medium.com/flutter/announcing-flutter-linux-alpha-with-canonical-19eb824590a9)announcing Linux as a first-class Flutter platform.

{% hint style="info" %}
Dart is the perfect language for building CLI apps and scripts
{% endhint %}

You can now build Linux desktop applications using Dart and Flutter. The fact that Canonical is redeveloping the Ubuntu installer in Flutter demonstrates their level of commitment to Flutter.

{% hint style="info" %}
Canonical is redeveloping the Ubuntu installer in Flutter.
{% endhint %}

If you are not familiar with Dart, it is a new language released by Google and it is now the [fastest growing ](https://www.linkedin.com/pulse/google-dart-tops-githubs-list-fastest-growing-2019-bill-detwiler)language on GitHub. Flutter is a cross platform graphical framework written in Dart supporting iOS, android, windows, Linux, Mac OS along with Linux arm support (yes it runs on your raspberry pi).

{% hint style="info" %}
Even if you have no intent of using Flutter you should look at Dart for building CLI apps.
{% endhint %}

As an old hack that started my career in C and 6502 assembler and has worked professionally with some dozen or more languages I like to say that 'Dart is Delightful'. It's an elegant language that brings simple solutions to common programming problems.

{% hint style="info" %}
Dart is Delightful to work with.
{% endhint %}

Dart is easy to learn and the development tooling is really easy to work with. It's often been called the love child of Java and JavaScript. It takes the best of these languages and removes the cruft.

At Noojee (the company I work for) we have 10s of thousands of lines of CLI code we use to support our production environment. This CLI code had been written in Bash, Perl, Ruby, Go, Rust, Python... In short, it was a mess and hard to maintain.

When we started working on a Flutter project we fell in love with Dart and saw a path to solve our maintenance problems with our CLI code.

Dart looked to be the perfect tool to replace all of our CLI apps and scripts. No more would we have to deal with archaic Bash and Perl scripts and no more magic Ruby code. We could convert all our scripts to Dart and use a single language for our production GUI and management tooling.

But the real pay off is that Dart is so simple to learn that any of our development team could help maintain CLI scripts and apps with almost no ramp time, try doing that with a Perl script.

Of course life is never as simple as it first looks.

### The Future is not so bright

Dart supports the concept of Futures. Futures are like Javascript Promises. Essentially a future tells the Dart VM that I'm going to do some work that will take a little while, so go and do something else and I will let you know when I'm done. Think of a Future as a super lightweight thread. A function that returns a Future is an async function.

Futures are great for a GUI app, particularly a mobile app, in that you need the GUI to be responsive even when you are fetching data or doing some large calculations.

The problem is that when writing a CLI app you really don't need to have a responsive UI and in fact Futures just make your life harder. Imagine the following code:

```dart
await createDir('/home/me');
await touch('/home/me/mything');
```

The createDir and touch functions are async functions which in Dart are implemented as Futures.

The 'await' statement tells dart to 'wait' for the function to finish before executing the next line.

Well, in a CLI application, just about every function call would need to be 'awaited' which is just tedious and gives zero benefits.

In fact in our early experiments with Dart, Futures were the cause of a multitude of disasters as it's very easy to forget to await each function (the latest dart linter does resolve this issue but at the time it was a significant issue).

Imagine in the above example if we had forgotten to await the createDir call. The result would be that the touch call would fail as the /home/me directory wouldn't exist as yet.

### waitFor to the rescue

The Dart CLI library has a solution for this; 'waitFor'. The waitFor command essentially tells Dart to change the async function (the Future) into a blocking function which is just perfect for CLI applications.

So we now had a path that made sense but we needed a common library for our team to share code.

## And DCli is born

And from that need was born DCli. DCli is a library of functions and classes designed specifically for creating CLI apps and scripts.

A founding principle of the libraries is that developers should never have to worry about Futures. Internally each of the DCli functions calls waitFor so that you don't need to think about futures. The above code simply becomes:

```dart
createDir('/home/me');
touch('/home/me/mything');
```

### Don't reinvent the wheel

One of the superpowers of Bash is that it makes it easy to call other CLI applications and process the output:

```bash
grep honda cars.txt | head > tophondas.txt
```

Whilst personally I despise Bash and its less than elegant syntax, you have to give it due credit for its ability to interact with other CLI apps.

In order to be able to replace our Bash scripts without re-inventing every linux app, it was going to be important that we were able to call existing CLI apps just as Bash does.

```dart
var hondas = ('grep honda carts.txt' | 'head').toList();

for (final honda in hondas)
{
    'tophondas.txt'.append(honda);
}
```

We view the DCli libraries ability to call external apps as so important that the library exposes more than a dozen methods for calling external apps and processing their output.

Here are some samples

```dart
'tail /var/log/syslog'.run;
'tail syslog'.start(workingDirectory: '/var/log', privileged: true);
var top = 'tail syslog'.firstLine;
```

### And the adventure begins

Over the past 18+ months the DCli library has grown into a sophisticated library providing all the tools required to build both simple and complex CLI applications.

DCli now consists of over 20K lines of code and internally Noojee now has over 100K lines of Dart/DCli running our production systems.

We have also developed a number of full blown apps using Dart and DCli:

#### Nginx-LE

A Docker container for Nginx with Lets Encrypt support baked in.

{% embed url="<https://github.com/bsutton/nginx-le>" %}

#### DSwitch

Switch between Dart channels (stable, beta, dev).

{% embed url="<https://github.com/bsutton/dswitch>" %}

#### DCli Scripts

A eclectic collection of scripts written in Dart and DCli

<https://github.com/bsutton/dcli_scripts>

## At the end of the day

Dart is a fantastic language and paired with DCli ,it really is the perfect language for building CLI apps and scripts.

Our dev and ops team love working with Dart and I think your team will too.

Dart and DCli are able to deliver all the pieces you require from a CLI development tool with none of the compromises.

Together Dart and DCli deliver

* Speed - Dart is fast
* Ease of learning - Dart is simple to learn, often described as the love child of Java and Javascript
* JIT or compiled - A Dart file can be run directly (JIT) or it can be compiled into a stand alone exe.
* Shebang support
* Large ecosystem of third party libraries vi a [pub.dev](https://pub.dev)
* Dart and DCli are Cross platform (Linux, Windows and OSX)
* Access to OS native system calls via [dart posix](https://pub.dev/packages/posix) and C libraries via [ffi](https://dart.dev/guides/libraries/c-interop)
* [Easy to install](https://dart.dev/get-dart)

If you want to give Dart and DCli a go I would recommend the following reading:

<https://dart.dev/get-dart>

[Dart Language Tour](https://dart.dev/guides/language/language-tour)

[Installing DCli](/getting-started)

[Writing your first CLI App](/writing-your-first-script)

## The full enchilada

Just in case you don't believe me regarding how easy it is. Here is a fully worked example:

```dart
sudo apt install dart
pub global activate dcli
dcli install
mkdir hello
cd hello
dcli create hello.dart
```

Copy the following text over the contents of hello.dart

```
#! /usr/bin/env dcli

import 'dart:io';
import 'package:dcli/dcli.dart';
import 'package:path/path.dart';

void main() {
  var name = ask('name:', required: true, validator: Ask.alpha);
  print('Hello $name');
  var pathToTestMe = join(HOME, 'testme');

  if (!exists(pathToTestMe))
  {
    createDir(pathToTestMe);
  }

  var pathToTxt = join(pathToTestMe, 'test.txt');
  pathToTxt.write('Hello $name');

  'cat $pathToTxt'.run;
}
```

And to run the script.

```dart
./hello.dart
```

Maybe you need better performance:

```dart
dcli compile hello.dart
./hello
```

Add hello to your path:

```dart
dcli compile --install --overwrite hello.dart
hello
```


# Improving your build environment

When building any pieces of software you invariable need to write software to help you write your software.

* build tools
* testing frameworks
* deployment tooling
* on production management systems

Historically these type of tools have been written in bash, powershell, ruby, python etc.

This is all well and good but as a Dart programmer life would be so much easier if you could write these tools in Dart.

Dart also has some significant advantages over each of the existing solutions.

* as single install includes all your build tools
* no dependency hell you typically see with python
* better performance
* ability to deploy an exe with no runtime required
* cross platform

This guide talks about how you can use Dart to improve you build environment and in particular what DCli brings to the table to make your life easier.

* [Existing tooling](/improving-your-build-environment/existing-tooling)
* [A home for our build tools](/improving-your-build-environment/a-home-for-your-build-tools)


# Existing tooling

As with any software project the first thing you should do is have a look at what pre-existing software is available that might solve your problem.

Maintaining any software is expensive, so if someone else is offering to do it for you...

Dart that eco system has a number of pre-exiting tools this guide notes a few of the more common ones.

{% hint style="info" %}
Drop us a line if you know of some other build tools that you think should be listed here.
{% endhint %}

### Native Dart tools

To be complete here are some of the build related tools built into dart

* dart format - format your source code
* dart doc - generates Dart api documenation
* dart fix - fixes common lint errors
* dart compile - compiles a Dart library with a `main` entry point to an exe
* dart create - creates a Dart project

### build\_runner

build\_runner is a core Dart package and used by many packages to automate the build process.

If you are doing json serialisation then you will already have come across build\_runner as it is used to generate the toJson and fromJson methods.

You can also use build\_runner in your on project to automate build steps.

Unfortunately the documentation on build runner is fairly sparse and and hard to follow. If you want to use build\_runner have a look at some of the projects that depend on build\_runner.

{% hint style="info" %}
pub.dev shows a list of packages that use (depend on) a package. This provides an easy way to find sample code for any package.
{% endhint %}

### Github Actions

Github supports actions which allow you to automate a build/test process each time you push to git.

Github actions support Linux, Windows and MacOS which allows you to build a target for multiple OSs.

Github actions are essentially a declarative system as opposed to procedural.

I've never been a fan of declarative build systems (all the way back to make) as they tend to rely on too many magic interactions between declarative steps.


# Building with Dart

As Dart developer there is no easier way to create your build, deploy and production environments than by using Dart.

In the rest of this article I wills the term 'build tools' to refer to the whole domain of build, deploy and production management tools.

Dart is also an excellent language for building console apps.

The DCli Console SDK was develop specifically for creating build tools and provides specific libraries to make the process easier.

Lets take a look at some of the tooling DCli provides to help you create build tools.

* DartSdk
* DartProject
* DartScript
* PubCache
* PubSpec


# A home for your build tools

Your approach to developing build tools should be like developing any piece of software. Build tools are a key part of a successful project and should not be neglected.

{% hint style="info" %}
Home is where heart is but I prefer my tools down in the shed.
{% endhint %}

## Where do my tools go?

There are five places I typically place build related tools depending on their scope.

{% hint style="info" %}
Include instructions on how to build your project in your README.MD
{% endhint %}

### Tool directory

The Dart specification includes a 'tool' directory under your project root. This is the right spot to place package specific tools.

```
bin
    flutter_main.dart
lib
    src
        some_flutter_code.dart
tool
    build.dart
```

If you need Dart additional Dart packages to build your tools you add them under the dev\_dependencies section of you Dart project.

```yaml
name: my_project
dependencies:
  some_flutter_package: ^1.0.0
dev_dependencies:
  dcli: ^1.14.0
```

The easiest way to add a dev dependency is from the cli

```bash
dart pub add --dev dcli
```

Dart libraries in you tool directory will normally contain a `main()` .

```dart
!# /bin/env dcli
void main(List<String> args)
{
}
```

You should have a library called 'build.dart' which is the script you run to build you project. Being consistent in the naming convention makes it easer for other users.

### Separate package in multi-package repo

For projects that are made up of several packages contained in a multi-project git repo I generally create an additional 'build' project in the root of the repo.

Using a separate package makes it easier for other users to find the build tools rather than looking in the tool directory of each package.

You may still have build tools in the tool directory of a specific package but these should related to build issues specific to that package.

When using a sperate build package you typically don't have any files in the tool directory of the build package (unless you need a build tool for the build package).

With a build package you place you apps in the `bin` directory of the build package and source code in the normal lib\src structure.

```
bin
   build.dart
lib
   src
      build_support_code.dart
```

The dependency for the build project are added to the normal dependencies section of the build projects pubspec.yaml.

```yaml
name: my_builder
dependencies:
  dcli:^1.14.0
```

### Common Build libraries

If you are running multiple projects with common build needs it can often be handy to create a 'build library' package. This package doesn't contain any main entry points.

Its sole purpose is to be a common repository of reusable code that your other build projects use.

```yaml
name: common_build_libs
dependencies:
    dcli: ^1.14.0
```

```
bin
    <empty>
lib
    src
        common_building_stuff.dart
```

### Build Tools Project(s)

Our deployment and production environments are built and run using Dart cli apps.

To support this environment we have a number of Dart packages that we using to deploy and run our systems.

We have a node management package that contains all of the cli apps required to create cloud instances (which refer to as nodes) and deploy our production systems to those nodes.

We then have a main package that is deployed to the system that provides any 'on node' management and diagnostic tools.

The deployment scripts compile the 'on node' scripts and deploy them as binaries to each node. (In this way we don't need to deploy Dart to the nodes reducing the security concerns around having a complete dev system on a production node).

We also have a number of specific build packages to build our Docker containers such as the Docker container we deploy on node to orchestrate backups.

### Tool Kit

Every developer has (or should have) a kit bag of tools that you take with you to help you get stuff done.

Dart makes packing your kit bag easy.

Create yourself a toolbag package and publish it to pub.dev.

Now where ever you go your tools are a simple command away

```
pub global activate my_toolbag
```

{% hint style="info" %}
To avoid polluting then pub.dev namespace please use a name that is unlikely to be used by a real project. I recommend using something of the nature \<mycompany\_toolbag>.

If you have an internal pub.dev repository then deploy your toolbag there.
{% endhint %}

The structure of your tool bag project should be:

```
bin
    build_all.dart
    pub_get_all.dart
    git_clean.dart
    docker_clean.dart
lib
    src
        common.dart
```

To expose each of the bin libraries as executables add an executables section to your pubspec.yaml

```yaml
name: mycompany_toolbag
dependencies:
    dcli: ^1.14.0
executables:
    build_all:
    pub_get_all:
    git_clean:
    docker_clean:
    
```

{% hint style="warning" %}
Don't leave any company sensitive information in your source code! When you publish to pub.dev all of you source code is published. Use .gitignore and .pubignore to exclude files.
{% endhint %}

To publish your package to pub.dev run:

```bash
pub publish
```

You will need a google account to publish.

Once published you can activate your new toolbag on any system that has Dart installed.

```bash
pub global activate mycompany_toolbak
```

Each of you executables listed in the `executables` section of the packages pubspec.yaml are now available to run from your PATH.

```bash
build_all
```

### Binary deployments from GIT

Sometimes you need your toolbag on systems that don't (and shouldn't) have the Dart SDK installed.

{% hint style="info" %}
It is poor practice to put the Dart SDK (or any SDK) on a production system as it significantly increases your security risk surface.
{% endhint %}

Being able to deploy a compiled Dart app (without needing a runtime) directly to a production system can solve this problem.

For the occasional use you can simply compile your Dart app to an executable and upload it to the production system.

```bash
dart compile exe bin/my_diag_tool.dart
or
dcli compile bin/my_diag_tool.dart
```

Then copy the resulting exe to the production system.

```bash
scp bin/my_diag_tool host.prodution.com:
```

Login to the remote system and you a ready to go.

If you regular access to you tools on production systems then you should consider deploying those tools as part of you production deployment process. But remember each time you deploy a tool to production you increase your risk profile.

### Use git as a binary repository

Another technique is to use git as a repository for your binaries.

Git allows you to create 'releases' which can include binary executables. You can then download these releases directly from Git onto any system.


# Olivier Revial - CLI apps made easy

{% embed url="<https://dev.to/stack-labs/cli-applications-made-easy-with-dart-dcli-8af>" %}


# Video: package of the week

Jermaine Oppong released a video with an overview of using DCli.

{% embed url="<https://www.youtube.com/watch?v=z99IxxWmD1Q&feature=youtu.be>" %}