The CLI Auto and CLI Auto Add-on apps provide a job engine (via a custom REST API endpoint) and an user interface to connect to several nodes (hosts) via ssh to execute and index (i.e. log to Splunk) the output of a Command Type (i.e. a set of CLI commands). Each Command Type configuration defines the set of CLI command(s), user input variables, data validation, success/failure regular expressions, and other configuration settings for the associated Command Type. Only a few Command Types were included in the initial version of the app; however, the app allows an user to develop and add more Command Types. With the wide support of CLI (ssh), the possible Command Types are numerous. It should be noted that the initial development of this app was geared toward user management CLI commands for Checkpoint, Palo Alto, and Sonicwall firewalls. The intent of this app is not to be a replacement for any management tools for these firewall brands or any other network device that supports CLI.
For CLI Auto versions 2.0.8 or greater, CLI Auto was split into 2 apps (i.e. the CLI Auto and CLI Auto Add-on apps) to add Splunk distributed/cluster environment features. So, it is required that both apps be installed for versions 2.0.8 or greater. Versions 2.0.8 or greater are not backwards compatible with versions 2.0.7 or older. For a single Splunk server environment, both the CLI Auto and CLI Auto Add-on apps are installed on the same server. For a distributed Splunk environment, it is recommended that the CLI Auto app is installed on a Search Head (SH) and/or a Deployer is used to deploy the CLI Auto app to the members of a Search Head cluster. In addition, the CLI Auto Add-on app is recommended to be installed on a Heavy Forwarder (HF) referred to as a Remote Addon Server in the CLI Auto app. As an example use case, the add features allow the CLI Auto Add-on app be installed on multiple HFs to run CLI commands on network nodes in multiple segmented networks which have the same subnets with the CLIAuto app being the common UI.
The CLI Auto and CLI Auto Add-on applications were developed on a Linux OS VM (CentOS) with a Splunk Dev environment. Due to limited resources, no development/testing at all was done against the Splunk Windows OS environment.
The Pexpect (pxssh) libraries (i.e. provide the ssh connection) were built for the CentOS hosting the Splunk dev environment and copied to the $SPLUNK_HOME/etc/apps/cliauto_addon/bin folder. The Openssh app (included with several Linux OS distros) is a dependency of the Pexpect (pxssh) libraries. And as of this writing, the ptyprocess libraries are a dependency for the Pexpect (pxssh) libraries; so the ptyprocess libraries were built for the OS hosting the Splunk dev environment and copied to the $SPLUNK_HOME/etc/apps/cliauto_addon/bin folder also. For most Linux OS Splunk installations, the /opt/splunk folder is the default value for the $SPLUNK_HOME environment variable.
The Splunk Python SDK libraries provide intergration to the Splunk environment for the job engine (reference the $SPLUNK_HOME/etc/apps/app_name/bin/splunklib and $SPLUNK_HOME/etc/apps/app_name/bin/utils folders - app_name is cliauto and cliauto_addon).
The Bootstrap (JS and CSS files) and Splunk Javascript SDK libraries help to provide front-end (UI) components (reference the $SPLUNK_HOME/etc/apps/cliauto/appserver/static folder).
The CLI Auto app was developed using the Chrome web browser. No testing/development was done with other web browsers.
The CLI Auto dashboard provides an user interface (UI) to start and monitor submitted jobs. To get the help notes for the UI, hover your cursor over the input fields and job record table headers to display the associated tooltip. Enter a Script User, enter a Script Password, select a Node List KVStore CSV Lookup file (see the instructions below to create the Node List file), and select a Command Type.
Select the Addon Server that will execute your job. For a single server environment, select the local Addon server. The Config dashboard can be used to configure a remote Addon Server(s) for the CLI Auto app. A remote Addon server is a Splunk Heavy Forwarder with the CLI Auto Add-on app installed. This input is available for CLI Auto app versions 2.0.8 or greater.
Enter the username that has permissions to login via to all of the nodes in the Node List. The tooltips are defined in the cliauto.conf (CLI Auto app) and cliauto_cmds.conf (CLI Auto Add-on app) files.
Enter the password for the Script username.
Select the CLI command(s) to be executed in the Command Type field. (example: SWFW Show Version)
The KVStore Lookup CSV file needs to contain a header record with 2 columns named host and ip_address. Please see the example in the screenshot below.
To create the KVStore Lookup CSV file within Splunk, click Settings->Lookups.
Click Add New for Lookup table files.
Select cliauto for the Destination app, click the Choose File button to select your Node List CSV file, give the Destination file an appropiate name, and click Save.
Verify that the KVStore Lookup table file was successfully saved.
Note: Splunk has other ways to create a KVStore Lookup file (a search command, other apps, etc.)
In the dashboard, select the KVStore CSV Lookup file that contains the nodes for the Command Type.
Click the Submit button to process the job.
Please type yes to confirm and submit job.
If you receive a popup similar to the ones below, your username may not have the proper permissions. Please see the Security section below with info to request your Splunk administrator for the proper access to the app.
The Status at the top of the dashboard and the status column of the job history table displays status info about the job. The Status at the top of the dashboard is the overall status of the job which includes the Addon server completing the job and the job events being searchable within Splunk. The status column of the job history table is the status of the Addon server completing the job.
To view the results for a job, click All, Success, or Fail to open the Splunk search app and execute the search with the filters provided in the respective url link. You can also find the job's results that were indexed to Splunk by creating you own search string (SPL) in the Splunk Search app with the index, source, sourcetype, and Job Id as filters. For example in the screenshot, index=main source=cliauto sourcetype=cliauto_ssh jobid=5bce1418e1382395b6480856. The default index, source and sourcetype for the app are index=main, source=cliauto and sourcetype=cliauto_ssh; so if you changed them in the CLI Auto app's conf file $SPLUNK_HOME/etc/apps/cliauto/default/cliauto.conf, your SPL search string will need to reflect your changes.
The "result" field in the job results is controlled by the success and failure regular expressions for the associated Command Type. The primary purpose of the "result" field is to provide a method to sort the job results to allow for an user to more quickly identify any possible next actions.
For ssh support, you need the ptyprocess and Pexpect (pxssh) python modules for this app, and are included in the installation files. However if needed, they can be downloaded from here and here.
Due to the limited module set installed with Splunk's Python instance, you may need to build these modules with a second installation of Python. Once you've built the modules you will need to copy the build/lib/ptyprocess and the build/lib/pexpect directories into your $SPLUNK_HOME/etc/apps/cliauto_addon/bin directory. Also, the Splunk Python SDK libraries need to exist in the $SPLUNK_HOME/etc/apps/cliauto_addon/bin directory.
Follow build instructions on the respective sites. No need to install. Instructions should be the same
for both modules:
cd $BUILD/pexpect-4.6.0
python setup.py build
cd $BUILD/ptyprocess-0.6.0
python setup.py build
Copy the respective directories to $SPLUNK_HOME/etc/apps/cliauto_addon/bin
cp -Rf $BUILD/pexpect-4.6.0/build/lib/pexpect $SPLUNK_HOME/etc/apps/cliauto_addon/bin
cp -Rf $BUILD/ptyprocess-0.6.0/build/lib/ptyprocess $SPLUNK_HOME/etc/apps/cliauto_addon/bin
Restart splunk.
The 5 diagrams below show the data flow, main execution path, and threads for the applications.
The CLI Auto app is secured with the cliauto_custom_endpoint capability and the cliauto_user role. It should be noted that the list_storage_passwords capability is included in the cliauto_user role to allow the app to read the password for the configured username/password credentials of a remote addon server. It is suggested in the interest of "least privilege" that the list_storage_passwords capability be disabled for the cliauto_user role if not needed. Contact your Splunk administrator to grant your Splunk user account the cliauto_user role to gain the required capabilities for the app.
Like any app, the level of security has a dependence on the environment used to host the app. So, it is suggested that the dependenices (see above) be updated periodically with their respective security patches/fixes.
For security reasons, it is recommended that the Custom (cli_custom) cmdtype be disabled in the cliauto_cmds.conf file unless needed for your use cases and proper security controls are in place. The intent of the Custom (cli_custom) cmdtype is to provide Splunk administrators/power users a tool to develop their own cmdtypes.
The data input validation for the app is to help guard against malicious and/or unintended uses. If your use case(s) requires data input that is blocked by the data input validation, the conf files may have a setting that can be modified to allow it. If not, you are welcome to suggest a new feature.
Some security related options of the ssh client (pexpect/pxssh library) are included in the cliauto.conf file, and should be reviewed/set for your environment/use cases. If more options are needed for your use case, a feature request is suggested.
It should be noted that the login function of the pexpect (pxssh) library was overridden for the app to account for specific use cases.
If you have suggestions to improve the security of the app, please contact support.
Using a Splunk VM on a laptop, the app has successfully executed several jobs with 300+ unique hosts and MaxThreads setting = 75 for multiple Command Types. A job with 1000 hosts (with non-active ip_addresses) was tested successfully also. Please let me know your feedback on the performance of the application.
It should be noted that the app allows for additional Command Types to be configured by Splunk Administrators/Power Users. The configuration of each included Command Type (cmdtype) can be found in the cliauto_cmds.conf file in the $SPLUNK_HOME/etc/apps/cliauto_addon/default folder. Each cmdtype configuration defines the CLI command(s), user input variables, data validation, success/failure regular expressions, and other configuration settings for the associated cmdtype. Per the "configuration file precedence" design feature of Splunk, it is recommended that new cmdtypes (or changes) be done by creating a $SPLUNK_HOME/etc/apps/cliauto_addon/local folder and conf file with the same filename (cliauto_cmds.conf). The contents of the new conf file must contain the stanza (aka cmdtype) along with the configuration setting. Also, please reference the cliauto_cmds.conf.spec file in the $SPLUNK_HOME/etc/apps/cliauto_addon/README folder for a description of the configuration settings.
restmap.conf file even if the python.version key is not defined for the stanza. The workaround was to add the python.version=python3 key/value pair to ALL stanzas in the restmap.conf file, and add the python.version key in a restmap.conf.spec file to prevent "Invalid key" messages from the btool app.Contact the developer- 3tressoftware@gmail.com
Per Splunk (JQuery 3.5+) requirement, added version=1.1 to dashboard tag in dashboard files
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.