Task: Download Warnings from an Analysis with a Windows Batch (.bat) File

You can use a script to download web GUI files from the command line.

This task provides a batch file for downloading warning reports from a specified analysis, along with some suggestions for modifying the script to suit your needs.

For other scripting options, see:

If you do not need the warning reports and just want a list of warnings, use codesonar dump_warnings.py instead.

To get all warnings from a single analysis, you can use the /analysis/analysis_id-allwarnings.xml URL. For example, to download all warnings from the analysis with ID 5 from the hub at http://myhub:7340, use http://myhub:7340/analysis/5-allwarnings.xml.

Permissions Required

The file requires that special user Anonymous has the following permissions for the analysis A of interest.

See Modifying the Batch File for information on modifying the file to specify credentials for a non-Anonymous user with the required permissions.

Other Requirements

Use the cURL shipped with CodeSonar: $CSONAR/third-party/curl/inst/bin/curl, where $CSONAR is the CodeSonar installation directory. Either:

The Download Batch File

The following file will download the HTML warning reports for:

and store them under the directory specified in the third argument to the file.

@ECHO off
set CALL_DIR=%CD%
set HUB=%1
set ANALYSIS_ID=%2
set SAVEDIR=%3

set ANALYSIS_CSV_URL="%HUB%/analysis/%ANALYSIS_ID%.csv"
set CURL_CMD=curl
if not exist "%SAVEDIR%" mkdir %SAVEDIR%
cd /D %SAVEDIR%
rm -fr *
%CURL_CMD% %ANALYSIS_CSV_URL% -o warnings.csv
for /F "skip=1 tokens=*" %%L in (warnings.csv) do call :handle_csv_line "%%L"
cd /D %CALL_DIR%
exit /b

:handle_csv_line
set LINE=%1
REM obtain URL component from last column of line
set LASTCOLUMN=
set /A COUNTER=0
:LOOP1BEGIN
set /A COUNTER+=1
for /F "tokens=%COUNTER% delims=," %%A in (%LINE%) do (
    if "%%A"=="" goto :LOOP1END
    set LASTCOLUMN=%%A
    goto :LOOP1BEGIN)
:LOOP1END
REM use URL component to compute URL and basename
set url2="%LASTCOLUMN:~0,-4%.html"
set /A COUNTER=0
:LOOP2BEGIN
set /A COUNTER+=1
for /F "tokens=%COUNTER% delims=/" %%A in ("%url2%") do (
    if "%%A"=="" goto :LOOP2END
    set basename=%%A
    goto :LOOP2BEGIN)
:LOOP2END
REM download the page
%CURL_CMD% %HUB%%url2% -o %basename%
exit /b

This batch file works as follows.

  1. It accesses the specified hub to download the CSV version of the Analysis: Warnings page for the analysis with the specified analysis ID.
    (This is the same file you would download if you navigated to the Analysis: Warnings page in the web GUI and clicked the CSV link. You may like to try this now so you can see how the file is formatted.)
  2. For each line in the CSV file (representing a single warning) except for the first (which is a header line), it does the following.
    1. Constructs the URL for the HTML warning report by taking the last entry in the line (representing the URL of the text version of the warning report) and replacing '.txt' at the end with '.html'.
    2. Downloads the URL, storing it in the location under the specified save directory that corresponds to its path.

Using the Batch File

To use this file with your hub, do the following.

  1. Create a directory to save your downloaded warnings in. The remainder of these instructions will refer to this directory as savedir.
  2. Copy download_warnings.bat to a suitable directory. The remainder of these instructions will refer to this directory as rundir.
    This should not be anywhere under savedir, since the file starts by deleting everything in that directory.
  3. Run the file:
    cd rundir
    download_warnings protocol://host:port aid savepath
    where
    protocol is the protocol for your hub: http or https.
    host:port is the location of your hub.
    aid is the analysis ID for the analysis whose warnings you wish to download.
    You can find the analysis ID:
    • in the URL specified at the end build/analysis command output, which will be of the form http://hub_location/analysis/analysis_id.html, or
    • in file path\to\projname.prj_files\aid.txt, where path\to\projname.prj_files\ is the analysis directory, or
    • by running the following command, where path\to\projname.prj_files\ is the analysis directory
      codesonar analysis_id.py path\to\projname.prj_files\
      or
    • in the Analysis Details section of the corresponding Analysis page.
    savepath is the path to the savedir directory you created in the first step.
    The file will output information about each HTML file it downloads.
  4. Inspect the contents of savedir. It should contain a file called warnings.csv, along with all the HTML warning reports from your specified analysis.
  5. Open one of the downloaded HTML files in a web browser to confirm that it contains a warning report.
    Note that links to files that the script did not download (including images) will be broken: this is expected behavior.

Invocation Examples

Troubleshooting

Get more verbose output For more verbose curl output, edit download_warnings.bat so that curl is invoked with the -v flag. For example: 
set CURL_CMD=curl -v
No files downloaded If the HTML warning reports are not present, check the command line output for information. If the only line of output is the URL of the Analysis: Warnings CSV file, this indicates that cURL did not attempt to download any warning reports. There are three possible reasons.
  • curl could not download the CSV file. The possible causes are as follows.
    • It is trying to download a page that does not exist.
      Check to make sure that you can open the CSV file URL in a web browser. You may need to pass different hub location or analysis ID arguments to the file.
    • It is trying to download a page that Anonymous does not have permission to access.
      If Anonymous does not have access to the Analysis: Warnings page,You will need to specify credentials for a user with the required permissions.
  • curl downloaded the CSV file, but the file did not list any warnings. The possible causes are as follows.
  • You have an HTTPS-enabled hub with a self-signed hub server certificate. To instruct curl to accept self-signed certificates, edit download_warnings.bat so that curl is invoked with the -k flag. For example: 
    set CURL_CMD=curl -k
Downloaded files contain "Permission Denied" messages If there are downloaded HTML files but they contain "Permission Denied" messages rather than warning reports, this indicates that Anonymous does not have ANALYSIS_WARNING_READ permission for the analysis. You will need to specify credentials for a user with the required permissions.

Modifying the Batch File

You may wish to make one or more of the following modifications.

Download warnings that satisfy a specific filter

If an Analysis: Warnings URL is specified without a query string component, the default warning filter setting for the authorizing user (Anonymous, in this case) is applied. To apply a different visibility filter, the URL must include a query string that specifies a filter value.

For example, suppose we want to specify the all visibility filter.

  1. Edit download_warnings.bat so that the line setting ANALYSIS_CSV_URL is: 
    set ANALYSIS_CSV_URL="%HUB%/analysis/%ANALYSIS_ID%.csv?filter=""all"""

Download XML warning reports instead

Warning reports can be output in text and XML formats as well as HTML. To download the XML versions, do the following.

  1. Edit download_warnings.bat so that the line immediately after :LOOPEND is changed to: 
    set url2=%LASTCOLUMN:~0,-4%.xml
    (That is, replace the occurrence of .html with .xml).
  2. If you want to process the downloaded files automatically, the XML schema at warning_report.xsd is likely to be useful.

Read analysis ID from analysis directory

Instead of specifying the analysis ID on the command line, you can change the file to read the analysis directory from the command line and then read the most recent analysis ID from the analysis directory.

  1. Edit download_warnings.bat so that the line setting ANALYSIS_ID is replaced with three lines: 
    set ANALYSIS_DIR=%2
set CSONAR=codesonar.exe
for /F "tokens=*" %%g in ('%CSONAR% analysis_id.py %ANALYSIS_DIR%') do (set ANALYSIS_ID=%%g)
    If it is not in your PATH, adjust the setting of CSONAR to include the full path to your codesonar.exe executable.
  2. When you invoke the script, specify the analysis directory as the second argument.

    For example: using the hub at http://[::1]:7341, download warnings for the analysis whose analysis directory is C:\myprojects\projectX.prj_files\ and save in directory c:\tmp\mywarnings.

    download_warnings http://[::1]:7341 C:\myprojects\projectX.prj_files\ c:\tmp\mywarnings

Provide credentials for a non-Anonymous user

If your hub is configured so that special user Anonymous does not have the required permissions, you will need to edit the file to submit credentials for a suitable hub user account.

We recommend using bearer authentication. Alternative mechanisms are described in the table below.

For bearer authentication, do the following.

  1. If you do not already have a suitable session and bearer token to use, generate them now.
    1. Navigate to the User Sessions page for your selected hub user account.
    2. Use the Create Session form to create a new session with a suitably long Expires setting.
      When you click Create Session, the page will be reloaded and the Bearer Token for your new session will be displayed at the top of the page.
      IMPORTANT: Make a note of the Bearer Token now. Once you refresh or navigate away from this page there will be no further opportunity to view the token.
    3. Save the bearer token to a file. The remainder of these instructions will refer to this file as path\to\bearerfile.
  2. Edit download_warnings.bat to add the following setting after the SAVEDIR setting. 
    set /p BEARER_TOKEN=<path\to\bearerfile
    where
    path\to\bearerfile is the path to the file containing the bearer token you want to use.
  3. Edit download_warnings.bat to modify the setting of CURL_CMD: 
    CURL_CMD=curl -H "Authorization: Bearer %BEARER_TOKEN%"

For more information about bearer authentication in CodeSonar, see User Sessions and Anonymous Sessions: Bearer Authentication.

If you don't want to use bearer authentication, you can choose one of the options from the following table.

Certificate If the hub is configured for certificate-based authentication, you can edit the file to specify a suitable user certificate.
  1. If the account does not already have a user certificate and key, generate them now.
  2. Edit download_warnings.bat to add the following settings after the SAVEDIR setting. 
    set CERT_PATH=path\to\usercert.pem
set KEY_PATH=path\to\certkey.pem
set CERT_AUTH=--cert %CERT_PATH% --key %KEY_PATH%
set POST_ARGS=-X POST -d "sif_sign_in=yes&sif_use_tls=yes&sif_log_out_competitor=yes"
    where
    path\to\usercert.pem is the path to the hub user account's user certificate.
    path\to\certkey.pem is the path to the private key corresponding to the user certificate.
  3. Edit download_warnings.bat to modify the setting of CURL_CMD:
    If your hub's hub server certificate is self-signed: 
    CURL_CMD=curl -k %POST_ARGS% %CERT_AUTH%
    Otherwise: 
    CURL_CMD=curl %POST_ARGS% %CERT_AUTH%
Hard-Coded Username/Password If you will be running the batch file under secure conditions, you may be willing to specify the account username and password directly in the file invocation.

For example, if your hub location is http://[::1]:7340 and the hub user account has username jean and password xyz123, the first argument to the file would be http://jean:xyz123@[::1]:7340.

Example: Use the hub user account with username jean and password xyz123 to authorize downloading warnings from the hub at http://[::1]:7340 for the analysis with ID 3, saving in directory c:\tmp\mywarnings:

download_warnings http://jean:xyz123@[::1]:7340 3 c:\tmp\mywarnings
Both username and password must be URL-encoded.
Username/Password: Other See the curl man page for alternative username/password authentication mechanisms.

See CodeSonar HTTP API: Authentication for more information on authentication strategies.

Writing Other Batch Files

You can follow the overall structure of this file to create batch files that download other kinds of file from the hub.

In general, the process for constructing a file will be along the following lines.

  1. Determine the GUI page type you are interested in.
  2. Look at the the GUI reference page for your required page type to determine the following information.
    • The page's URL or URL scheme: use this to construct the URL or URLs for your file to download.
    • The alternative page output formats that are available: if you want to process pages in one of these formats rather than HTML, check that your required format is available and construct the download URLs accordingly.
    • The RBAC permissions required to access the page and its contents. If special user Anonymous does not have these permissions, the batch file will need to provide authentication credentials for a hub user account that has the permissions.
  3. If your script will be carrying out a task that modifies the hub database (adding, deleting, or modifying elements), inspect the GUI page HTML to look at the <form ... > element that provides access to the functionality you are interested in. Your script will need to include an HTTP POST request (via curl or similar) that corresponds to the one issued when the form is submitted.
  4. Make sure your batch file includes the following elements.
    • One or more download URLs, or a mechanism for constructing them.
    • Some way of handling the downloaded URLs: one of the following.
      • The location to which the URLs should be downloaded, or a way to obtain the location.
      • An explicit redirect to the null location.
      • No specific handling, so that the downloaded files are all part of the script output.
    • A curl invocation that acts on the the URL or URLs.
  5. See the troubleshooting notes above if you encounter problems.
  6. This task uses a batch file in order to be accessible to as many Windows users as possible. If you have access to PowerShell, you will probably prefer it over batch files for scripted hub interactions.
    You may also be interested in using Python scripts to interact with the hub: if you have no local Python installation, you can use the cspython executable shipped with CodeSonar.

Running Batch Files Automatically

You can use your system tools to arrange for the batch file to be run automatically.

For example, you may choose to use the Windows Task Scheduler.

Links

Note. This page contains references to HTTP API documentation, which is served directly by the hub and cannot be accessed via a file:// URL. For active HTTP API documentation links, start a hub (if one is not already running), then open the manual from the hub.