Introduction
The next post in this series on static code analysis will present a tool that will automate the Veracode scan submission, report download, and analysis. I am first demonstrating how one can perform the scans manually.
After logging into https://web.analysiscenter.veracode.com/login/#/login
Click on start a static scan. This gets you into the wizard
.
Next, you must upload your packaged file — they have a cheat sheet to help with packaging instructions.
The next task is to pick the file to upload using the big “Select Files” button.
One would have to click next to Validate Upload and scan. Another key point is that after this screen, the next screen will take some time to complete – since we have asked it to immediately scan.
Veracode will complete the scan, and the results are available by selecting the latest report in the left-hand sidebar.
Above I am showing the manual scan; all of this does take a bit of time as the instance on the cloud performs the actual scan job.
A programmatic approach
Introducing the Veracode API which helps automate things. It is a good idea to try to find a library that wraps the API. The Veracode Community GitHub offers a python package that can assist with calling the Veracode API. The library’s GitHub may be found here.
Let’s look at the top, high-level code of what we are trying to do, uploading, scanning, and reporting.
# Note: for this to work you must have your api credentials
# in the file ~/.veracode/credentials
def main():
args = check_arguments()
application = args.application[0]
file = args.file[0]
use_json = args.json
use_mongo = args.mongo
app_info = get_application_info(application)
upload_file(app_info['id'], file)
build_id = begin_pre_scan(app_info['id'])
wait_for_report_to_be_ready(app_info['id'], build_id)
summary_report = get_summary_report(app_info['guid'])
findings_report = get_findings_report(app_info['guid'])
print_summary_analysis(summary_report, findings_report, use_json, use_mongo)
if args.download:
download_report(build_id, file)
return 0
Let us decode what is happening here. First, we obtain the application information and start the pre-scan. Secondly, we wait for the cloud to complete its task. Thirdly, we obtain the findings report and summary report. If you give the —json flag, Print Summary Analysis will also create a json file. The program always prints the report to the terminal. The program writes the JSON record into a MongoDB when the —mongo flag is present. The database only works if you have a running MongoDB server. You also need the environment variable MONGO_CONNECTION_STRING for the database to function. The program will create the database, and populate any necessary collections, and insert the record.
Getting the application IDs, there is a legacy id and a guid id of the Application is important for the internal workings. Application in the Veracode world is a ‘bucket’ in which you drop some project code. An application represents one or more real-world we are planning on scanning. Here’s how we get the application id with the Veracode library
def get_application_info(application_name):
api = veracode_api_py.Applications()
result = {}
response = None
try:
response = api.get_by_name(application_name)
except veracode_api_py.VeracodeAPIError as e:
print(EXCEPTION_DETAILS_FOLLOW)
print(e)
sys.exit(1)
if len(response) > 0:
result['guid'] = response[0]['guid']
result['id'] = response[0]['id']
return result
else:
print("Call to get applications failed")
sys.exit(1)
The use of the library has the advantage of getting support for retries and handling API rate limitations. Veracode API limits the number of calls to its API to six API calls per minute per account.
The next code snippet shows how we can upload the target file to Veracode with the library:
def upload_file(app_id, file_name):
print('Uploading file to Veracode: ' + file_name + '...')
api = veracode_api_py.VeracodeAPI()
try:
api.upload_file(app_id, file_name)
except veracode_api_py.VeracodeAPIError as e:
print(EXCEPTION_DETAILS_FOLLOW)
print(e)
sys.exit(1)
The next step is to begin the scan. Veracode provides a legacy XML API for some functions and a more modern REST API for others. Right now, the library supports both, but at the time of writing, one API, https://analysiscenter.veracode.com/api/5.0/beginprescan.do one is missing. The code below shows how one can implement this.
def begin_pre_scan(app_id):
# the library does not support this call, we roll our own
print('Initiating scan...')
endpoint = 'https://analysiscenter.veracode.com/api/5.0/beginprescan.do'
try:
response = requests.post(endpoint,
params={'app_id': app_id, 'auto_scan': 'true'},
auth=RequestsAuthPluginVeracodeHMAC(),
headers=HEADERS)
except requests.RequestException as e:
print("Call to " + endpoint + " failed.")
print(EXCEPTION_DETAILS_FOLLOW)
print(e)
sys.exit(1)
if response.ok:
if 'has been updated by another transaction' in response.text:
print('Another scan is likely in progress for ' + app_id + '. Please try again later.')
sys.exit(1)
root = element_tree.fromstring(response.text)
return root.attrib['build_id']
print("Call to " + endpoint + " failed.")
sys.exit(1)
Waiting for the scan to finish is easy, it’s supported in the library.
def wait_for_report_to_be_ready(application_id, build_id):
date_time = datetime.now().strftime("%H:%M:%S")
print('Starting to wait for the report at: ' + date_time)
results_ready = False
while not results_ready:
results_ready = check_if_build_is_ready(application_id, build_id)
if not results_ready:
time.sleep(60)
date_time = datetime.now().strftime("%H:%M:%S")
print('Waiting... ' + date_time)
After this step and once we’re done waiting, we can either download the pdf or make a REST call to get the both summary and findings. Moreover, here is the summary:
def get_summary_report(application_guid):
print('Getting summary report...')
api = veracode_api_py.SummaryReport()
response = None
try:
response = api.get_summary_report(application_guid)
except veracode_api_py.VeracodeAPIError as e:
print(EXCEPTION_DETAILS_FOLLOW)
print(e)
sys.exit(1)
if len(response) > 0:
return response
print("Call to get summary report failed")
sys.exit(1)
And here I present the code to get the findings:
def get_findings_report(application_guid):
print('Getting findings report...')
api = veracode_api_py.Findings()
try:
response = api.get_findings(application_guid, request_params={'scan_type': 'STATIC'})
except veracode_api_py.VeracodeAPIError as e:
print(EXCEPTION_DETAILS_FOLLOW)
print(e)
sys.exit(1)
if len(response) > 0:
return response
print('Call to get findings report failed.')
sys.exit(1)
After this, decoding the json takes a bit of work — my program provides both a terminal report and a json document.
def report_summary(summary_report_json, use_json, report_json, use_mongo):
result = {}
severities = summary_report_json['severity']
for severity in severities:
categories = severity['category']
if len(categories) > 0:
for category in categories:
category_severity = category['severity']
if category_severity in result:
result[category_severity] = result[category_severity] + category['count']
else:
result[category_severity] = category['count']
print('Printing summary analysis...')
print('Application Name: ' + str(summary_report_json['app_name']))
print('Analysis Id: ' + str(summary_report_json['analysis_id']))
print('Account Id: ' + str(summary_report_json['account_id']))
print('Application Id: ' + str(summary_report_json['app_id']))
print('Static analysis unit id: ' + str(summary_report_json['static_analysis_unit_id']))
print('Flaws not mitigated: ' + str(summary_report_json['flaws_not_mitigated']))
print('Policy compliance status: ' + str(summary_report_json['policy_compliance_status']))
print('Rating: ' + str(summary_report_json['static-analysis']['rating']))
print('Score: ' + str(summary_report_json['static-analysis']['score']))
if use_json or use_mongo:
report_json['application_name'] = summary_report_json['app_name']
report_json['analysis_id'] = str(summary_report_json['analysis_id'])
report_json['account_id'] = str(summary_report_json['account_id'])
report_json['application_id'] = str(summary_report_json['app_id'])
report_json['static_analysis_unit_id'] = str(summary_report_json['static_analysis_unit_id'])
report_json['flaws_not_mitigated'] = str(summary_report_json['flaws_not_mitigated'])
report_json['policy_compliance_status'] = summary_report_json['policy_compliance_status']
report_json['rating'] = str(summary_report_json['static-analysis']['rating'])
report_json['score'] = str(summary_report_json['static-analysis']['score'])
# more but I will leave that out, you get the gist
It is then easy to write to the mongo database. Doing this with python is easier than using mongo in .net.
def write_to_mongo(report_json):
client = pymongo.MongoClient(os.environ.get('MONGO_CONNECTION_STRING'))
db = client["veracode"]
collection = db["veracode_reports"]
record = {"name": report_json['application_name'], "value": report_json}
collection.insert_one(record)
And consequently, here’s what the console output looks like:
Initiating scan... Uploading file... Starting to wait for the report at: 23:35:43 Waiting... 23:36:44 Waiting... 23:37:45 Waiting... 23:38:47 Getting summary report... Getting findings report... Printing summary analysis... Application Name: Redacted Analysis Id: 21932616 Account Id: 91009 Application Id: 1575585 Static analysis unit id: 21948266 Flaws not mitigated: 3 Policy compliance status: Did Not Pass Rating: A Score: 99 Number of very high severity issues: 0 Number of high severity issues: 0 Number of medium severity issues: 2 Number of low severity issues: 0 Number of informational severity issues: 1 Veracode found the following issues: ------------------------------------------------------ Sequential Issue Number: 0 Description: The application contains hard-coded information that may contain credentials to an external service. The use of hard-coded credentials significantly increases the possibility that the account being protected will be compromised. ?uid=devops?password= Store credentials out-of-band from the application code. Follow best practices for protecting credentials stored in locations such as configuration or properties files. References: CWE issue_id: 1 cwe: Use of Hard-coded Credentials cwe id: 798 severity: 3 file_path: bitbucket-build-approver/controllers/mergedprcontroller.cs file_line_number: 190 module: bitbucket-build-approver.dll ------------------------------------------------------ Sequential Issue Number: 1 Description: Standard random number generators do not provide a sufficient amount of entropy when used for security purposes. Attackers can brute force the output of pseudorandom number generators such as rand(). If this random number is used where security is a concern, such as generating a session key or session identifier, use a trusted cryptographic random number generator instead. These can be found on the Windows platform in the CryptoAPI or in an open source library such as OpenSSL. In Java, use the SecureRandom object to ensure sufficient entropy. References: CWE issue_id: 2 cwe: Insufficient Entropy cwe id: 331 severity: 3 file_path: bitbucket-build-approver/controllers/mergedprcontroller.cs file_line_number: 367 module: bitbucket-build-approver.dll ------------------------------------------------------ Sequential Issue Number: 2 Description: The program fails to release or incorrectly releases the variable 5__2, which was previously allocated by a call to system_runtime_dll.System.IO.StreamReader.!newinit_0_1(). Ensure that all code paths properly release this resource. References: CWE issue_id: 3 cwe: Improper Resource Shutdown or Release cwe id: 404 severity: 0 file_path: bitbucket-build-approver/rawjsonbodyinputformatter.cs file_line_number: 21 module: bitbucket-build-approver.dll Software Composition Analysis - Potentially Vulnerable Libraries No components found. Writing report to veracode_report.json Writing report to MongoDB End of report.
The JSON report
{
"generation_time": "2022-11-07T23:55:20.000113",
"application_version": "scan-and-report v1.0",
"application_name": "redacted",
"analysis_id": "21932616",
"account_id": "91009",
"application_id": "1575585",
"static_analysis_unit_id": "21948266",
"flaws_not_mitigated": "3",
"policy_compliance_status": "Did Not Pass",
"rating": "A",
"score": "99",
"medium": "2",
"informational": "1",
"findings": [
{
"id": "1",
"description": "The application contains hard-coded information that may contain credentials to an external service. The use of hard-coded credentials significantly increases the possibility that the account being protected will be compromised. ?uid=devops?password= Store credentials out-of-band from the application code. Follow best practices for protecting credentials stored in locations such as configuration or properties files. References: CWE",
"issue_id": "1",
"cwe": "Use of Hard-coded Credentials",
"cwe_id": "798",
"severity": "3",
"file_path": "bitbucket-build-approver/bitbucket-build-approver/controllers/mergedprcontroller.cs",
"file_line_number": "190",
"module": "bitbucket-build-approver.dll"
},
{
"id": "2",
"description": "Standard random number generators do not provide a sufficient amount of entropy when used for security purposes. Attackers can brute force the output of pseudorandom number generators such as rand(). If this random number is used where security is a concern, such as generating a session key or session identifier, use a trusted cryptographic random number generator instead. These can be found on the Windows platform in the CryptoAPI or in an open source library such as OpenSSL. In Java, use the SecureRandom object to ensure sufficient entropy. References: CWE",
"issue_id": "2",
"cwe": "Insufficient Entropy",
"cwe_id": "331",
"severity": "3",
"file_path": "bitbucket-build-approver/bitbucket-build-approver/controllers/mergedprcontroller.cs",
"file_line_number": "367",
"module": "bitbucket-build-approver.dll"
},
{
"id": "3",
"description": "The program fails to release or incorrectly releases the variable 5__2, which was previously allocated by a call to system_runtime_dll.System.IO.StreamReader.!newinit_0_1(). Ensure that all code paths properly release this resource. References: CWE",
"issue_id": "3",
"cwe": "Improper Resource Shutdown or Release",
"cwe_id": "404",
"severity": "0",
"file_path": "bitbucket-build-approver/bitbucket-build-approver/rawjsonbodyinputformatter.cs",
"file_line_number": "21",
"module": "bitbucket-build-approver.dll"
}
]
}
And there we have it, the basics of automating Veracode static scans with python. The next post in the series is:
Part 6: Storing the scan results in a mongo database
