General

codesonar bi_transfer.py: Transfer Warning Information to a Business Intelligence (BI) Tool

The codesonar bi_transfer.py command can transfer warning information directly from the hub to your organization's business intelligence (BI) tool.

With CodeSonar factory settings, the only supported BI tool is Qlik. If you want to upload to a different tool, contact CodeSecure support.

Overview

The codesonar bi_transfer.py command transfers warning information from the CodeSonar hub to your local business intelligence (BI) tool, without the need to interact with the CodeSonar Web GUI or write a custom script.

There are four steps.

  1. Create a configuration file for your BI tool.
    This step only needs to be performed once.
  2. Check the connection to your BI tool.
    This step is optional, though highly recommended.
  3. Download a CSV file of warning information from the hub.
  4. Upload the CSV file to your BI tool.

This page provides a full description of the codesonar bi_transfer.py command and all its options.

bi_transfer.py is a codesonar Python subcommand.

Command Line

There are four forms of the command line for transferring warning information to your BI tool, representing the four steps of the process.

1. Create configuration file
codesonar bi_transfer.py --create-bi-conf path/to/conffile.conf \
--bi-tool tool_name
2. Check connection
codesonar bi_transfer.py --test-connection path/to/conffile.conf
3. Download from hub
codesonar bi_transfer.py --download path/to/wfile.csv \
--type warning --hub [protocol://]host:port --search wsearch_expr \
[-auth authtype] [-hubuser username] [-hubpwfile pwfile] [-hubbearerfile bearerfile] [-hubcert certfile] [-hubkey privatekeyfile]
or
codesonar bi_transfer.py --download path/to/wfile.csv \
--type warning --analysis-url aurl [--search wsearch_expr] \
[-auth authtype] [-hubuser username] [-hubpwfile pwfile] [-hubbearerfile bearerfile] [-hubcert certfile] [-hubkey privatekeyfile]
4. Upload to BI tool
codesonar bi_transfer.py --upload path/to/wfile.csv \
--bi-config path/to/conffile.conf --validation path/to/tmpdir/ \
[--remote-file-name-prefix pref]

1. Create configuration file

codesonar bi_transfer.py --create-bi-conf path/to/conffile.conf \
--bi-tool tool_name
--create-bi-conf path/to/conffile.conf Create a configuration file at the specified location.
Depending on the value specified for --bi-tool, you may be prompted for additional information about your BI tool.
This configuration file will subsequently be used in the "Check connection" and "Upload to BI tool" steps.
--bi-tool toolname Specify the BI tool you are using.
With CodeSonar factory settings, the only available setting is --bi-tool qlik, representing the Qlik data analytics tool.
If you want to upload to a different BI tool, contact CodeSecure support.

2. Check connection

codesonar bi_transfer.py --test-connection path/to/conffile.conf
--test-connection path/to/conffile.conf Test the connection to the BI tool identified in the specified configuration file.

3. Download from hub

codesonar bi_transfer.py --download path/to/wfile.csv \
--type warning --hub [protocol://]host:port --search wsearch_expr\
[-auth authtype] [-hubuser username] [-hubpwfile pwfile] [-hubbearerfile bearerfile] [-hubcert certfile] [-hubkey privatekeyfile]

or

codesonar bi_transfer.py --download path/to/wfile.csv \
--type warning --analysis-url aurl [--search wsearch_expr] \
[-auth authtype] [-hubuser username] [-hubpwfile pwfile] [-hubbearerfile bearerfile] [-hubcert certfile] [-hubkey privatekeyfile]
--download path/to/wfile.csv Download information from the hub to a CSV file at the specified location.
--type warning The only kind of information that can currently be downloaded from the hub is warning information.
--analysis-url aurl Get warnings from the analysis whose Analysis:Warnings page URL is aurl.
  • If specified multiple times, only the last aurl is used.
  • The set of retrieved warnings can be further refined with --search.
--hub [protocol://]host:port The hub location.
--search wsearch_expr
 Get only warnings that satisfy the constraints in wsearch_expr.
  • wsearch_expr must be a warning search language term. If wsearch_expr contains spaces or other special characters, it must be double-quoted.
  • If specified multiple times, only the last wsearch_expr is used.
  • If --analysis-url is specified, the search scope is the specified analysis. Otherwise, the search scope is all analyses on the hub.
[-auth authtype],
[-hubuser username],
[-hubpwfile pwfile],
[-hubbearerfile bearerfile],
[-hubcert certfile],
[-hubkey privatekeyfile] 		Specify how CodeSonar should authenticate the download request. For details, see Hub Authentication: Authenticated codesonar Subcommands.
If you are using codesonar bi_transfer.py in a script or a similar unattended context, it will probably be most convenient to use bearer authentication: specify -auth bearer -hubbearerfile bearerfile, where bearerfile contains a valid session bearer token.

4. Upload to BI tool

codesonar bi_transfer.py --upload path/to/wfile.csv \
--bi-config path/to/conffile.conf --validation path/to/tmpdir/ \
[--remote-file-name-prefix pref]
--upload path/to/wfile.csv Upload information from the specified CSV file to the BI tool identified by --bi-config.
--bi-config path/to/conffile.conf The configuration file identifying the BI tool to upload to.
--validation path/to/tmpdir/ If the CSV file at path/to/wfile.csv is larger than 495 MB it will be split into chunks for upload. Use this option to specify the temporary directory to be used to store these chunks, along with any temporary files associated with validating the files.
  • The directory must already exist.
  • The chunks and temporary files associated with validation will be deleted. (The CSV file specified with --upload is not deleted.)
[--remote-file-name-prefix pref] The uploaded chunks will be given names of the form pref_chunknum_timestamp.csv, where
  • pref is specified with this option. If not specified, the prefix is file.
  • chunknum is the (0-based) chunk index.
    • If --upload path/to/wfile.csv does not need to be split into chunks, there will be a single uploaded file: pref_0_timestamp.csv.
    • Otherwise, there are multiple uploaded files: pref_0_timestamp.csv, pref_1_timestamp.csv,...
  • timestamp is the timestamp, and is the same across all chunks generated for a given command instance.

Using codesonar bi_transfer.py with Qlik

The following instructions describe how to configure and use codesonar bi_transfer.py to transfer warning information from your CodeSonar hub to your Qlik deployment.

  1. Sign into your Qlik user account.
  2. Make a note of your Qlik hostname, including protocol. It will be something like https://tenantid.region.qlikcloud.com.
  3. Get a Qlik API key.
    1. Open your Qlik user Profile Settings page (from the user account menu at the top right).
    2. Switch to the API Keys tab.
    3. Click Generate new key.
      A Generate new API key dialog will open.
    4. Enter a useful description (for example, codesonar-key) and expiry period where prompted, then click Generate.
      Your new API key will be displayed.
    5. Copy and save the API key. You will need it to set up your configuration file.
    If the API Keys tab is not present, or it is present but the Generate new key button is disabled, you will need to enable the Developer role for your user account (from the Management Console) and then select Enable API keys from the API Keys section of the Settings tab. If you encounter difficulties, consult the Qlik documentation or discuss with your local Qlik administrator.
  4. Choose the Qlik space where you will store the uploaded warning information.
  5. Switch to your regular command line for the remainder of these steps.
  6. Change to a suitable working directory.
  7. Create a configuration file for working with your Qlik deployment. In this example, we will save the configuration file directly in the working directory, in file qlik_auth.conf.
    codesonar bi_transfer.py --create-bi-conf qlik_auth.conf --bi-tool qlik
    Provide configuration information when prompted:
    host the Qlik hostname you identified in step 2.
    Auth Key the Qlik API Key you created in step 3.
    Space the Qlik space you identified in step 4. If the space does not already exist, it will be created.
  8. Test the connection to the BI tool.
    codesonar bi_transfer.py --test-connection qlik_auth.conf
    If your configuration file is correctly set up and your Qlik deployment is reachable, this will output a confirmation message. Otherwise, it will fail with an error message: use the error message to diagnose your problem. If necessary, regenerate your configuration file with corrected information.
  9. Download warning information from the hub.
    Suppose we want all the warning information from the analysis with Analysis ID 5, and our hub is located at http://myhub.example.com:7340. To download this information to file warnings5.csv in the current working directory, use the following command line.
    codesonar bi_transfer.py --download warnings5.csv --type warning \
    --analysis-url http://myhub.example.com:7340/analysis/5.html
    A confirmation message will be printed on success.
  10. Upload the information from warnings5.csv to your Qlik instance.
    mkdir tmp
    codesonar bi_transfer.py --upload warnings5.csv --bi-config qlik_auth.conf \
    --validation ./tmp/ --remote-file-name-prefix w5
    A confirmation message will be printed on success.
  11. Go to Qlik to inspect the uploaded files (you may need to switch to the Qlik space you specified in the configuration step).

The warning information is now uploaded to Qlik and ready for you to apply your own data analytics.