Skip to main content

Demisto SDK

test The Demisto SDK is a Python library designed to aid the development process, both to validate entities being developed and to assist in the interaction between your development setup and Cortex XSOAR or Cortex XSIAM. This guide will help you get acquainted with the Demisto SDK, including installation and set up and will provide some basic information about key commands to aid you in the development process. For additional information, please see our full Demisto SDK documentation.

Note that in some cases fetching remote files is needed. In the case of a private repository and an un-configured DEMISTO_SDK_GITHUB_TOKEN environment variable, files will be fetched from the remote branch of the local repo.

Installation and Setup#

The Demisto SDK library supports Python 3.8 and above. Before installing, please make sure you have a compatible Python version installed and active. The Demisto SDK is designed to work on macOS and Linux machines. Windows users are encouraged to run Demisto-SDK commands through a dev-container, WSL or a docker image. See here for more information.

If you have followed our development setup bootstrap process you don't need to install the SDK manually as it is installed for you on your venv.

These installation steps are only required if you are not working with the bootstrap or if you are working on a repository which is not part of the Content repository. To manually install the Demisto-SDK, enter your terminal and run the command:

pip3 install demisto-sdk

After running the command the library should install - as a first step, check what version of the SDK you are using, by running:

demisto-sdk -v

This command should print the version you are using. You can check the latest released version here.

If you already have the SDK installed and you wish to upgrade the version, run the following command:

pip3 install --upgrade demisto-sdk

If you would like to contribute to the Demisto SDK project, check out the Contributing to XSOAR SDK guide.

Environment Variable Setup#

Some SDK commands require you to have an interaction with the Cortex XSOAR or Cortex XSIAM server. Examples of such interactions include uploading and downloading entities to or from XSOAR or XSIAM and running commands in the CLI.

To use these functions, set up the required environment variables:

For Cortex XSOAR 6.x Only#

  1. Get your API key by going to the Cortex XSOAR server -> Settings -> Integrations -> API Keys -> Get Your Key -> Give your key a name and press Generate Key.

  2. Copy the given key.

  3. Add the following parameters to ~/.zshrc and ~/.bash_profile:

    export DEMISTO_BASE_URL=<http or https>://<demisto-server url or ip>:<port>
    export DEMISTO_API_KEY=<API key>

    For example:

    export DEMISTO_BASE_URL=http://127.0.0.1:8080
    export DEMISTO_API_KEY=XXXXXXXXXXXXXXXXXXXXXX

    Alternatively, you can use a .env file in the root repository with the above variables.

For Cortex XSIAM or Cortex XSOAR 8.x and above#

  1. The base URL should be retrieved from the XSIAM instance by navigating to Settings -> Configurations -> API Keys -> clicking on the Copy URL button located at the top right corner of the page, and not the browser URL.

  2. The API key should be set to a Standard security level and must have the Instance Administrator role.

  3. The XSIAM_AUTH_ID environment variable needs to be set. Get it from the ID column of the created API Key.

  4. Add the following parameters to ~/.zshrc and ~/.bash_profile:

    export DEMISTO_BASE_URL=<BASE_URL>
    export DEMISTO_API_KEY=<API_KEY>
    export XSIAM_AUTH_ID=<THE_XSIAM_AUTH_ID>

    For example:

    export DEMISTO_BASE_URL=https://api-xsiam-server.us.paloaltonetworks.com
    export DEMISTO_API_KEY=XXXXXXXXXXXXXXXXXXXXXX
    export XSIAM_AUTH_ID=1

    Alternatively, you can use a .env file in the root repository with the above variables.

Please note that once the XSIAM_AUTH_ID environment variable is set, the SDK commands will be configured to work with a Cortex XSIAM / XSOAR 8.x instance. In order to set Demisto SDK to work with a Cortex XSOAR 6.x instance, you need to delete the XSIAM_AUTH_ID parameter from your environment. To do this, please run the following command:

unset XSIAM_AUTH_ID

Using the SDK in Private Repositories#

Finally, if you are using a private GitHub repository, some SDK functions require an interaction with Git. To use the SDK you should setup your GitHub token. To generate your token on GitHub use the following guide and then run:

export DEMISTO_SDK_GITHUB_TOKEN=<token>

General CLI Usage#

Run SDK commands from the CLI by following this basic structure:

demisto-sdk <command_name> <command_arguments>

If you are not sure of the available commands, run:

demisto-sdk -h

This command gives you a full list of available commands as well as a short description of each command's purpose. If you need the list of arguments available for a command, run:

demisto-sdk <command_name> -h

This gives you a full list of the command's arguments and their descriptions. We add new arguments and features regularly, so please feel free to approach us with new ideas and suggestions!

Run using Docker image#

You can run the Demisto-SDK using a docker image. For more details go to Demisto-SDK Docker.

Commands#

This guide will give you a short tutorial about the basic SDK commands. While there are additional commands and use cases found in the SDK, these are the main commands you should know to begin:

init#

Use this command to create a new pack, integration or a script.

Examples and Use Cases#

  • Create a new pack:

    demisto-sdk init -n myNewPackName

    This creates a new pack under the Packs directory with the name myNewPackName. After the pack initialization ends the command gives you the option to create a new integration in the pack.

  • Create a new integration in a specified directory:

    demisto-sdk init --integration -n myIntegration -o Packs/myNewPack/Integrations

    This creates a new integration with the name myIntegration under the directory Packs/myNewPack/Integrations.

  • Create a new feed integration:

    demisto-sdk init --integration -n myFeed -t FeedHelloWorld

    This creates a new integration named myFeed which will be based on the FeedHelloWorld integration. The init command has several preset templates that you can choose as the basis for your integration or script. By default the template is the BaseIntegration and BaseScript but you can also use HelloWorld, HelloIAMWorld, or FeedHelloWorld for integrations and HelloWorldScript for scripts. These templates have more robust integration information and examples. Additionally, some templates are used to create the basic structure for feed and IAM integrations.

  • Create a new script:

    demisto-sdk init --script -n myScript

    This command creates a new script named myScript.

For additional information see here.

validate#

Use this command to make sure your .yml, .json and .md files are up to Cortex XSOAR or Cortex XSIAM standards.

Examples and Use Cases#

  • Validate all committed files:

    demisto-sdk validate

    This command identifies all the files that were committed via git in your current branch and runs validations on them. Note that this is the variation of the command that runs in our build process, in case of a build failure run this to recreate it locally.

  • Validate all committed and staged files:

    demisto-sdk validate -g

    This command identifies all the files that were committed and staged via git in your current branch and runs validations on them.

  • Validate a specific file:

    demisto-sdk validate -i Packs/myPack/Integrations/myIntegration/myIntegration.yml

    This runs validation only on the file Packs/myPack/Integrations/myIntegration/myIntegration.yml. This variation does not detect the file status via git and thus will not check for backwards compatibility changes.

  • Validate all committed and staged files, print ignored files and errors and debug git:

    demisto-sdk validate -g --print-ignored-errors --print-ignored-files --debug-git

    This command validates all committed and staged files and prints out additional information which is hidden by default:

    • Which files were ignored.
    • Error codes that were ignored.
    • A report on which files were identified and in what status via git. This additional information might be helpful in debugging SDK commands or figuring out why a file did not validate.

Validate Error Structure and Ignoring Errors#

Each one of our validation errors follows a basic structure:

File_Path: [Error_Code] - Error Message

For example:

Packs/CortexXDR/Integrations/CortexXDRIR/CortexXDRIR.yml: [BA100] - The version for our files should always be -1, please update the file.

At the end of the validation a summary appears with the file paths and error codes.

The error codes serve two main functions:

  1. The first two letters are used to identify the type of error encountered. For example, DO is a Docker related error, BC is backwards compatibility related and ST is a yml/json structure error. A full list of abbreviations can be found in the full validation documentation.

  2. The error code can be used to ignore specific error types. To ignore an error go to the file .pack-ignore file and input the following structure:

    [file:file_name]
    ignore=<error_code_to_ignore>

    For example:

    [file:CortexXDRIR.yml]
    ignore=IN126,IN135
    • Note: Not all error codes can be ignored! It is preferable to fix errors rather than ignore them. Please consult with a Cortex XSOAR or XSIAM team member before ignoring an error.

For additional information see here.

lint#

Use this command to make sure your .py and .ps1 files are up to Cortex XSOAR or XSIAM standards. This command runs several libraries to validate your code, including:

  • Flake8 - Makes sure your code is up to PEP8 standards.
  • MyPy - Validates type annotations, assignments and additional Python checks.
  • Vulture - Finds unused code.
  • Bandit - Finds security issues.
  • XSOAR Linter - An internal linter used to identify specific restrictions and provide best practice advice for your code.
  • Pylint and Pytest - Used to run your attached unit test (in the _test.py) file on the integration/script's Docker and will report your code coverage.
  • PowerShell test and analyze - Code linters for PowerShell.

At the end of the command a short report appears with all of the error and warnings found, as well as failed and passed unit tests.

For additional information please see our documentation about linting and unit testing.

Examples and Use Cases#

  • Run lint on all committed and changed files:

    demisto-sdk lint -g

    This command identifies all the files that were committed and staged via git in your current branch and runs lint on them. Note that this is the variation of the command that runs in our build process, in case of a build failure run this to recreate it locally.

  • Run lint without flake8 on a specific file:

    demisto-sdk lint -i Packs/myPack/Integrations/myIntegration/ --no-flake8

    This runs the lint command on the integration found in Packs/myPack/Integrations/myIntegration/ - please note that the path requested is to the directory not to the file itself. Also, it should be noted that there are additional flags used to turn off any specific linter such as --no-mypy and --no-xsoar-linter, etc.

For additional information see here.

secrets#

Use this command to find secrets such as emails and IP addresses in your files. Cortex XSOAR is an open source product, its code can be found on a public repository on GitHub and thus is very visible. You have a responsibility to identify and eliminate any secrets before they get to our repository or even to our pull requests.

Please note: this command is not fool proof and a manual review of the files is still highly recommended.

Examples and Use Cases#

  • Detect secrets in your files:

    demisto-sdk secrets

    This detects secrets in all your changed files. Please note that this command can have some false positives. You can make the command less sensitive by adding the -ie flag.

Ignoring secrets#

At times there is information that might be flagged incorrectly by the command as a secret. Or it may be "secret" but we wish to share it publicly (for example the support email address). To ignore a specific secret, enter it to the packs's .secrets-ignore file. For example:

https://myurl.com
https://my2ndUrl.com

For additional information see here.

format#

Use this command to format an XSOAR or XSIAM entity in accordance with our standards. In some cases, when a file is downloaded from the XSOAR server, it might contain additional fields that are not required when entering the entity to the content repository. The format command will remove the unnecessary fields and make any fixes needed to the existing fields.

Examples and Use Cases#

  • Format a specific file:

    demisto-sdk format -i Packs/myPack/Integrations/myIntegration/myIntegration.yml

    This formats the file Packs/myPack/Integrations/myIntegration/myIntegration.yml in accordance with our standards.

  • Format a pack and update script and integration Docker images:

    demisto-sdk format -i Packs/myPack -ud

    This formats all files in myPack. It also updates any Docker images in integrations and scripts in the pack.

For additional information see here.

upload#

Use this command to upload an entity to a Cortex XSOAR or Cortex XSIAM. Be sure to set up the required environment variables prior to running this command in order to establish a connection between demisto-sdk and the XSOAR or the XSIAM server.

Examples and Use Cases#

  • Upload an integration to the server:

    demisto-sdk upload -i Packs/myPack/Integrations/myIntegration

    This uploads the integration found in Packs/myPack/Integrations/myIntegration to the preset server.

  • Upload a pack to the server without certificate validation:

    demisto-sdk upload -i Pack/myPack --insecure

    This iterates over all the content entities in the pack myPack and uploads them to the preset server without checking the certification. Note that this command does not upload the entities as a whole pack but instead uploads them individually.

  • Zip and upload a whole pack to the Marketplace:

    demisto-sdk upload -i Pack/myPack -z

    This will zip, upload and install the pack myPack in the preset server's Marketplace.

    When uploading to the Marketplace, the command will overwrite any pack which already exists in the server. In order to avoid mistakes, a prompt will appear asking you to approve the upload.

For additional information see here.

zip-packs#

Use this command to generate zipped packs and upload them to the Marketplace in Cortex XSOAR or Cortex XSIAM. Be sure to set up the required environment variables prior to running this command in order to establish a connection between demisto-sdk and the XSOAR or the XSIAM server.

Examples and Use Cases#

  • Zip a pack:

    demisto-sdk zip-packs -i Campaign -o Packs/Campaign

    This will zip the Campaign pack into Campaign.zip file in the Packs/Campaign directory.

  • Zip and upload a whole pack to the Marketplace:

    demisto-sdk zip-packs -i Campaign -o Packs/Campaign -u

    This will zip, upload and install the pack Campaign in the preset server's Marketplace.

    When uploading to the Marketplace, the command will overwrite any pack which already exists in the server. In order to avoid mistakes, a prompt will appear asking you to approve the upload.

For additional information see here.

download#

Use this command to download entities from a Cortex XSOAR or XSIAM server to your local repository. Be sure to set up the required environment variables prior to running this command in order to establish a connection between demisto-sdk and the server.

This command can be useful when developing within the Cortex XSOAR or Cortex XSIAM server itself and downloading the new entities to your local environment in order to continue with the contribution process.

Notes and Limitations#

  • JavaScript integrations and scripts are not downloadable using this command.
  • If there are files that exist both in the output directory and are specified in the input, they will be ignored. To override this behavior so that existing files will be merged with their newer version, use the --force/-f flag.
  • For consistency, it is assumed that for each integration or script the folder containing it will have the same name as the integration/script name with no separators. For example the integration Test Integration_Full-Name, will be under ~/.../Packs/TestPack/Integrations/TestIntegrationFullName/.

Examples and Use Cases#

  • See which files are downloadable:

    demisto-sdk download -lf

    This lists all the files which are downloadable using this command from the preset server. Note: Do not run the -lf flag with -i or -o.

  • Download a file to a given pack:

    demisto-sdk download -i "My Integration" -o Packs/myPack

    This downloads the entity named My Integration and places it in the appropriate directory within myPack. Note that if My Integration exists in the pack, it will not be downloaded.

  • Download several files to a given pack and overwrite any file which already exists:

    demisto-sdk download -i "My Integration" -i myScript -o Packs/myPack -f

    This downloads both the My Integration and myScript entities to myPack. If any of the files already exist in the pack they are overwritten.

  • Download all available custom files to a given pack:

    demisto-sdk download -a -o Packs/myPack

    This downloads all accessible files (listed with the -lf flag) to the appropriate directories in myPack.

For additional information see here.

update-release-notes#

Use this command to automatically generate release notes for a given pack and update the pack_metadata.json version.

Running this command creates a new release notes file under the ReleaseNotes directory in the given pack in the form of X_Y_Z.md where X_Y_Z is the new pack version. After running this command, add the newly created release notes file to GitHub and add your notes under their respective headlines. For example:

#### Integrations
##### Cortex XDR - IOC
- Fixed an issue where searching more than 10K indicators failed when using ElasticSearch.
- Updated the Docker image to: *demisto/python3:3.9.4.18682*.
#### Incident Fields
- **XDR Similar Incidents** - New incident field.
#### Playbooks
##### Cortex XDR incident handling v3
- Added a new machine learning script to search for similar incidents by shared incident fields and indicators.
#### Layouts
##### Cortex XDR Incident
- Updated layout with a new section for similar incidents.

Further information about how to run this command can be found here.

generate-docs#

Use this command to generate a readme file for your integration, script or playbook.

Running this command creates a new README.md file in the same directory as the entity on which it ran, unless otherwise specified using the -o flag. To generate command examples, set up the required environment variables prior to running this command in order to establish a connection between demisto-sdk and the server, as well as create a file containing some command examples to be run for the documentation.

Further information about how to run this command can be found here.

generate-yml-from-python#

Use this command to generate a YML file from Python code that includes its special syntax.

The output file name will be the same as the Python code with the .yml extension instead of .py.

The generation currently supports integrations only. The feature is supported from content Base pack version 1.20.0 and on.

Example run:

demisto-sdk -i my_integration.py -v

Further information about the syntax and how to run this command can be found here.

Setting a Preset Custom Command Configuration#

You can create your own configuration for the demisto-sdk commands by creating a file named .demisto-sdk-conf within the directory from which you run the commands. This file will enable you to set a default value to the existing command flags that will take effect whenever the command is run. This can be done by entering the following structure in the file:

[command_name]
flag_name=flag_default_value

Note: Be sure to use the flag's full name and use _ instead of a - if it exists in the flag name (e.g. instead of no-docker-checks use no_docker_checks).

Here are a few examples:

  • As a user, I would like to not use the MyPy linter in my environment when using the lint command. In the .demisto-sdk-conf file I'll enter:

    [lint]
    no_mypy=true

  • As a user, I would like to include untracked git files in my validation when running the validate command. In the .demisto-sdk-conf file I'll enter:

    [validate]
    include_untracked=true

  • As a user, I would like to automatically use minor version changes when running the update-release-notes command. In the .demisto-sdk-conf file I'll enter:

    [update-release-notes]
    update_type=minor
Edit this page
Last updated on
Report an Issue