Task: Download Warnings from an Analysis with a Shell Script

You can use a script to download web GUI files from the command line.

This task provides a shell script for downloading warning reports from a specified analysis, along with some suggestions for modifying the script to suit your needs.

For other scripting options, see:

If you do not need the warning reports and just want a list of warnings, use codesonar dump_warnings.py instead.

To get all warnings from a single analysis, you can use the /analysis/analysis_id-allwarnings.xml URL. For example, to download all warnings from the analysis with ID 5 from the hub at http://myhub:7340, use http://myhub:7340/analysis/5-allwarnings.xml.

Permissions Required

The script requires that special user Anonymous has the following permissions for the analysis A of interest.

See Modifying the Script for information on modifying the script to specify credentials for a non-Anonymous user with the required permissions.

Other Requirements

The example script uses the sed and tail utilities.

Use the cURL shipped with CodeSonar: $CSONAR/third-party/curl/inst/bin/curl, where $CSONAR is the CodeSonar installation directory. Either:

The Download Script

The following script will download the HTML warning reports for:

and store them under the directory specified in the third argument to the script.

#! /bin/sh -e

if [ $# -ne 3 ]; then
    echo "Usage: $0 HUB ANALYSIS_ID SAVEDIR"
    exit 1
fi

HUB=$1
ANALYSIS_ID=$2
SAVEDIR=$3
ANALYSIS_CSV_URL="${HUB}/analysis/${ANALYSIS_ID}.csv"
CURL_CMD=curl
rm -fr "$SAVEDIR"
mkdir "$SAVEDIR"
cd "$SAVEDIR"

fetch(){
   "$CURL_CMD" "$@"
}

fetch "${ANALYSIS_CSV_URL}" -o warnings.csv
for url in $(tail -n +2 < warnings.csv | sed -e 's/^.*,//')
do
    url2=$(echo "$url" | sed 's/[.]txt/.html/')
    fetch "${HUB}${url2}" -o "$(basename "$url2")"
done

This shell script works as follows.

  1. It accesses the specified hub to download the CSV version of the Analysis: Warnings page for the analysis with the specified analysis ID.
    (This is the same file you would download if you navigated to the Analysis: Warnings page in the web GUI and clicked the CSV link. You may like to try this now so you can see how the file is formatted.)
  2. For each line in the CSV file (representing a single warning) except for the first (which is a header line), it does the following.
    1. Constructs the URL for the HTML warning report by taking the last entry in the line (representing the URL of the text version of the warning report) and replacing '.txt' at the end with '.html'.
    2. Downloads the URL, storing it in the location under the specified save directory that corresponds to its path.

Using the Script

To use this script with your hub, do the following.

  1. Create a directory to save your downloaded warnings in. The remainder of these instructions will refer to this directory as savedir.
  2. Copy download_warnings.sh to a suitable directory. The remainder of these instructions will refer to this directory as rundir.
    This should not be anywhere under savedir, since the script starts by deleting everything in that directory.
  3. Run the script:
    cd rundir
    ./download_warnings.sh protocol://host:port aid savepath
    where
    protocol is the protocol for your hub: http or https.
    host:port is the location of your hub.
    aid is the analysis ID for the analysis whose warnings you wish to download.
    You can find the analysis ID:
    • in the URL specified at the end build/analysis command output, which will be of the form http://hub_location/analysis/analysis_id.html, or
    • in file path/to/projname.prj_files/aid.txt, where path/to/projname.prj_files/ is the analysis directory, or
    • by running the following command, where path/to/projname.prj_files/ is the analysis directory
      codesonar analysis_id.py path/to/projname.prj_files/
      or
    • in the Analysis Details section of the corresponding Analysis page.
    savepath is the path to the savedir directory you created in the first step.
    The script will output information about each HTML file it downloads.
  4. Inspect the contents of savedir. It should contain a file called warnings.csv, along with all the HTML warning reports from your specified analysis.
  5. Open one of the downloaded HTML files in a web browser to confirm that it contains a warning report.
    Note that links to files that the script did not download (including images) will be broken: this is expected behavior.

Invocation Examples

Troubleshooting

Get more verbose output For more verbose curl output, edit download_warnings.sh so that curl is invoked with the -v flag. For example: 
fetch(){
   "$CURL_CMD" -v "$@"
}
No files downloaded If the HTML warning reports are not present, check the command line output for information. If the only line of output is the URL of the Analysis: Warnings CSV file, this indicates that cURL did not attempt to download any warning reports. There are three possible reasons.
  • curl could not download the CSV file. The possible causes are as follows.
    • It is trying to download a page that does not exist.
      Check to make sure that you can open the CSV file URL in a web browser. You may need to pass different hub location or analysis ID arguments to the script.
    • It is trying to download a page that Anonymous does not have permission to access.
      If Anonymous does not have access to the Analysis: Warnings page,You will need to specify credentials for a user with the required permissions.
  • curl downloaded the CSV file, but the file did not list any warnings. The possible causes are as follows.
  • You have an HTTPS-enabled hub with a self-signed hub server certificate. To instruct curl to accept self-signed certificates, edit download_warnings.sh so that curl is invoked with the -k flag. For example: 
    fetch(){
  "$CURL_CMD" -k "$@"
}
Downloaded files contain "Permission Denied" messages If there are downloaded HTML files but they contain "Permission Denied" messages rather than warning reports, this indicates that Anonymous does not have ANALYSIS_WARNING_READ permission for the analysis. You will need to specify credentials for a user with the required permissions.

Modifying the Script

You may wish to make one or more of the following modifications.

Download warnings that satisfy a specific filter

If an Analysis: Warnings URL is specified without a query string component, the default warning filter setting for the authorizing user (Anonymous, in this case) is applied. To apply a different visibility filter, the URL must include a query string that specifies a filter value.

For example, suppose we want to specify the all visibility filter.

  1. Edit download_warnings.sh so that the line setting ANALYSIS_CSV_URL is: 
    ANALYSIS_CSV_URL='$HUB/analysis/${ANALYSIS_ID}.csv?filter="all"'

Download XML warning reports instead

Warning reports can be output in text and XML formats as well as HTML. To download the XML versions, do the following.

  1. Edit download_warnings.sh so that the first line of the do loop is changed to: 
    url2=$(echo "$url" | sed 's/[.]txt/.xml/')
    (That is, replace the occurrence of .html with .xml).
  2. If you want to process the downloaded files automatically, the XML schema at warning_report.xsd is likely to be useful.

Read analysis ID from analysis directory

Instead of specifying the analysis ID on the command line, you can change the script to read the analysis directory from the command line and then read the most recent analysis ID from the analysis directory.

  1. Edit download_warnings.sh so that the line setting ANALYSIS_ID is replaced with three lines: 
    ANALYSIS_DIR=$2
CSONAR_CMD=codesonar
ANALYSIS_ID=$("$CSONAR_CMD" analysis_id.py "$ANALYSIS_DIR" | tr -d '\r')
    If it is not in your PATH, adjust the setting of CSONAR_CMD to include the full path to your codesonar executable.
  2. When you invoke the script, specify the analysis directory as the second argument.

    For example: using the hub at http://[::1]:7341, download warnings for the analysis whose analysis directory is /myprojects/projectX.prj_files/ and save in directory /tmp/mywarnings.

    ./download_warnings.sh http://[::1]:7341 /myprojects/projectX.prj_files/ /tmp/mywarnings

Provide credentials for a non-Anonymous user

If your hub is configured so that special user Anonymous does not have the required permissions, you will need to edit the script to submit credentials for a suitable hub user account.

We recommend using bearer authentication. Alternative mechanisms are described in the table below.

For bearer authentication, do the following.

  1. If you do not already have a suitable session and bearer token to use, generate them now.
    1. Navigate to the User Sessions page for your selected hub user account.
    2. Use the Create Session form to create a new session with a suitably long Expires setting.
      When you click Create Session, the page will be reloaded and the Bearer Token for your new session will be displayed at the top of the page.
      IMPORTANT: Make a note of the Bearer Token now. Once you refresh or navigate away from this page there will be no further opportunity to view the token.
    3. Save the bearer token to a file. The remainder of these instructions will refer to this file as path/to/bearerfile.
  2. Edit download_warnings.sh to add the following setting after the SAVEDIR setting. 
    BEARER_TOKEN=$(cat path/to/bearerfile)
    where
    path/to/bearerfile is the path to the file containing the bearer token you want to use.
  3. Change the definition of fetch() to: 
    fetch(){
  "$CURL_CMD" -H "Authorization: Bearer ${BEARER_TOKEN}" "$@"
}

For more information about bearer authentication in CodeSonar, see User Sessions and Anonymous Sessions: Bearer Authentication.

If you don't want to use bearer authentication, you can choose one of the options from the following table.

Certificate If the hub is configured for certificate-based authentication, you can edit the script to specify a suitable user certificate.
  1. If the account does not already have a user certificate and key, generate them now.
  2. Edit download_warnings.sh to add the following settings after the SAVEDIR setting. 
    CERT_PATH=path/to/usercert.pem
KEY_PATH=path/to/certkey.pem
    where
    path/to/usercert.pem is the path to the hub user account's user certificate.
    path/to/certkey.pem is the path to the private key corresponding to the user certificate.
  3. Change the definition of fetch() to: 
    fetch(){
  "$CURL_CMD" -k -X POST -d "sif_sign_in=yes&sif_use_tls=yes&sif_log_out_competitor=yes" --cert "$CERT_PATH" --key "$KEY_PATH" \
      "$@"
}
    Omit the -k argument if your hub's hub server certificate is not self-signed.
Hard-Coded Username/Password If you will be running the shell script under secure conditions, you may be willing to specify the account username and password directly in the script invocation.

For example, if your hub location is http://[::1]:7340 and the hub user account has username jean and password xyz123, the first argument to the script would be http://jean:xyz123@[::1]:7340.

Example: Use the hub user account with username jean and password xyz123 to authorize downloading warnings from the hub at http://[::1]:7340 for the analysis with ID 3, saving in directory /tmp/mywarnings:

./download_warnings.sh http://jean:xyz123@[::1]:7340 3 /tmp/mywarnings
Both username and password must be URL-encoded.
Username/Password: Other See the curl man page for alternative username/password authentication mechanisms.

See CodeSonar HTTP API: Authentication for more information on authentication strategies.

Writing Other Scripts

You can follow the overall structure of this script to create shell scripts that download other kinds of file from the hub.

In general, the process for constructing a script will be along the following lines.

  1. Determine the GUI page type you are interested in.
  2. Look at the the GUI reference page for your required page type to determine the following information.
    • The page's URL or URL scheme: use this to construct the URL or URLs for your script to download.
    • The alternative page output formats that are available: if you want to process pages in one of these formats rather than HTML, check that your required format is available and construct the download URLs accordingly.
    • The RBAC permissions required to access the page and its contents. If special user Anonymous does not have these permissions, the shell script will need to provide authentication credentials for a hub user account that has the permissions.
  3. If your script will be carrying out a task that modifies the hub database (adding, deleting, or modifying elements), inspect the GUI page HTML to look at the <form ... > element that provides access to the functionality you are interested in. Your script will need to include an HTTP POST request (via curl or similar) that corresponds to the one issued when the form is submitted.
  4. Make sure your shell script includes the following elements.
    • One or more download URLs, or a mechanism for constructing them.
    • Some way of handling the downloaded URLs: one of the following.
      • The location to which the URLs should be downloaded, or a way to obtain the location.
      • An explicit redirect to the null location.
      • No specific handling, so that the downloaded files are all part of the script output.
    • A curl invocation that acts on the the URL or URLs.
  5. See the troubleshooting notes above if you encounter problems.
  6. The shell script in this task is designed to be as portable as possible. If more sophisticated control constructs and shell tools are available on your system, you can use them to streamline your hub interaction scripts.
    You may also be interested in using Python scripts to interact with the hub: if you have no local Python installation, you can use the cspython executable shipped with CodeSonar.

Running Scripts Automatically

You can use your system tools to arrange for the shell script to be run automatically.

For example, if you are using cron, add the following line to your crontab to run download_warnings.sh at 2:05am every day, downloading from http://red:7341 the warnings issued by the analysis whose analysis directory is /home/projectX.prj_files/ and saving them in /tmp/mywarnings. 
5 2 * * * /path/to/download_warnings.sh http://red:7341 `cat /home/projectX.prj_files/aid.txt` /tmp/mywarnings
If you have modified the script to take different arguments, specify those arguments accordingly.

Links

Note. This page contains references to HTTP API documentation, which is served directly by the hub and cannot be accessed via a file:// URL. For active HTTP API documentation links, start a hub (if one is not already running), then open the manual from the hub.