debian-cve-check: Improve error handling when dst.json download fails

[teradat]

Purpose of pull request

If the Debian Security Tracker json (dst.json) download fails or file is invalid, the CVE check cannot be performed.
In this case, output backtrace [1] because there is insufficient error checking.
So improve to error handling.

[1]

File: '<path-to>/meta-debian/classes/debian-cve-check.bbclass', lineno: 33, function: debian_cve_check
     0029:            _pkg_file_name = os.path.basename(_pkg_uri)
     0030:            pkgname = _pkg_file_name.split(";")[0].split("_")[0]
     0031:            break
     0032:
 *** 0033:    if pkgname not in dst_data.keys():
     0034:        bb.note("%s is not found in Debian Security Tracker." % pkgname)
     0035:        return
     0036:
     0037:    deb_patched, deb_unpatched = deb_check_cves(d, dst_data[pkgname])
Exception: AttributeError: 'NoneType' object has no attribute 'keys'

Background

When cve-check cannot be performed, there are two ways of thinking depending on the purpose of bitbake:

1. The purpose is build, want to continue to build
   e.g. bitbake core-image-minimal
2. The purpose is cve-check, want to immediately terminate with an error
   e.g. bitbake bash -c cve_check

Add "CVE_CHECK_ERROR_ON_FAILURE" variable to satisfy these wants.

• Set "0" (is default): skip the CVE check and continue with build of bitbake
  By disabling "CVE_CHECK_DB_FILE" variable, CVE check will be skipped in Poky's do_cve_check() function.
  This is the same behavior as if the NVD database download failed in Poky, skip the CVE check and continue with build.
• Set "1": bitbake return fatal error immediately
  Immediately exit with bb.fatal().

Details of improvements

The following changes in this commit:

• Add exception handling to load_json()
  Delete file exist check as they are handled by exception handling.
• Add check timestamp of the dst.json file
  (Even if the dst.json download fails,) A successfully downloaded dst.json file may still exist, so if the timestamp within CVE_DB_UPDATE_INTERVAL (default 24 hours), it is considered a valid file.
• Add error handling logic for "CVE_CHECK_ERROR_ON_FAILURE" variable
  Change the log output lebel to match behavior of this variable.
• Some code style fixes
  Add spaces after comma.

Test

How to test

1. local.conf setting

   • Set DEBIAN_SECRUTY_TRACKER_JSON_URL to empty so that dst.json download fails
   • If DEBIAN_SECRUTY_TRACKER_JSON_URL_append exists, comment it out.

   Add the following to local.conf.

   MACHINE = "qemuarm64"
   INHERIT += " cve-check debian-cve-check"
   #DEBIAN_SECRUTY_TRACKER_JSON_URL_append = " <URL>"
   DEBIAN_SECRUTY_TRACKER_JSON_URL = ""
   

   And, for purpose of test, modify CVE_CHECK_ERROR_ON_FAILURE.

   • For skip behavior

     CVE_CHECK_ERROR_ON_FAILURE = "0"
     

   • For fatal behavior

     CVE_CHECK_ERROR_ON_FAILURE = "1"
     

2. Preparing for testing
   Remove the dst.json file.

   $ rm <path-to>/downloads/CVE_CHECK/DEBIAN/dst.json
   

3. Build bash package and check log

   $ bitbake -v bash; echo "bitbake return-code: $?"
   

Test result

For skip behavior

Displayed WARNING message and bitbake succeeds.

build$ bitbake -v bash; echo "bitbake return-code: $?"
...snip...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: CVE database recently updated, skipping
NOTE: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: Download json file from 
WARNING: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: Invalid URL (unknown url type: '')
WARNING: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: DST database download failed
NOTE: bash-5.0-r0 do_cve_check: debian_cve_check: No DST database found, skipping CVE check
NOTE: bash-5.0-r0 do_cve_check: No CVE database found, skipping CVE check
NOTE: Tasks Summary: Attempted 2024 tasks of which 2021 didn't need to be rerun and all succeeded.

Summary: There were 2 WARNING messages shown.
bitbake return-code: 0

For fatal behavior

By bb.fatal(), displayed ERROR message and bitbake stops.

build$ bitbake -v bash; echo "bitbake return-code: $?"
...snip...
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: CVE database recently updated, skipping
NOTE: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: Download json file from 
WARNING: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: Invalid URL (unknown url type: '')
WARNING: cve-update-nvd2-native-1.0-r0 do_populate_cve_db: DST database download failed
ERROR: bash-5.0-r0 do_cve_check: debian_cve_check: No DST database found
ERROR: bash-5.0-r0 do_cve_check: 
ERROR: bash-5.0-r0 do_cve_check: Function failed: debian_cve_check
ERROR: Logfile of failure stored in: <path-to>/build/tmp-glibc/work/aarch64-emlinux-linux/bash/5.0-r0/temp/log.do_cve_check.3005243
ERROR: Task (<path-to>/build/../repos/meta-debian/recipes-debian/bash/bash_debian.bb:do_cve_check) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2023 tasks of which 2021 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  <path-to>/build/../repos/meta-debian/recipes-debian/bash/bash_debian.bb:do_cve_check
Summary: There were 2 WARNING messages shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
bitbake return-code: 1