[TRACKER] collection test failures

[gotmax23]

(All the collections that have been retired from Ansible 10.3.0 or otherwise no longer have sanity test failures in that version should be checked off.)

./main.py -i data/9/ansible-9.1.0-sanity.yaml -t build/ansible-build-data/9/ansible-9.1.0-tags.yaml -u data/9/ansible-9.1.0-missing-upstreams.yaml failures --bullet="- [ ]" --reverse | wl-copy

• cisco.intersight -- Community package requirements: sanity tests CiscoDevNet/intersight-ansible#108
  15512 validate-modules
• fortinet.fortimanager -- Community package requirements: sanity tests fortinet-ansible-dev/ansible-galaxy-fortimanager-collection#80, Community package requirements: sanity tests fortinet-ansible-dev/ansible-galaxy-fortimanager-collection#93
  13015 validate-modules
• cisco.ucs -- Community package requirements: sanity tests CiscoDevNet/ansible-ucs#40
  9502 validate-modules
• ovirt.ovirt (upstream runs the tests but has their own script to preprocess their sources which we cannot run here)
  136 WRONG_HASH
  113 yamllint
  66 validate-modules
  60 compile-python-3.10
  60 compile-python-3.11
  60 compile-python-3.12
  59 compile-python-2.7
  59 compile-python-3.6
  59 compile-python-3.7
  59 compile-python-3.8
  59 compile-python-3.9
  11 ansible-doc
  1 WRONG_GALAXY_YML_NAME
• community.network -- Community package requirements: sanity tests ansible-collections/community.network#591 (exclusion discussion started in https://forum.ansible.com/t/should-we-remove-community-network-from-the-ansible-community-package/8030)
  654 BANNED_SANITY
• check_point.mgmt -- Community package requirements: sanity tests and repository management CheckPointSW/CheckPointAnsibleMgmtCollection#121, Community package requirements: sanity tests and repository management CheckPointSW/CheckPointAnsibleMgmtCollection#142
  240 validate-modules
  120 WRONG_HASH
  3 ansible-doc
• netapp_eseries.santricity -- Community package requirements: sanity tests NetApp/santricity#23
  342 validate-modules
  3 ansible-doc
  2 yamllint
• azure.azcollection -- Community package requirements: sanity tests ansible-collections/azure#1417
  311 validate-modules
• google.cloud -- (Removal from Ansible) Community package requirements: sanity tests ansible-collections/google.cloud#613
  186 validate-modules
• theforeman.foreman -- Community package requirements: sanity tests theforeman/foreman-ansible-modules#1703 (upstream runs the tests but has their own script to preprocess their sources which we cannot run here)
  153 yamllint
• wti.remote -- Community package requirements: sanity tests and CI wtinetworkgear/wti-collection#15, Community package requirements: repository management wtinetworkgear/wti-collection#17
  85 validate-modules
  7 ansible-doc
  4 yamllint
• hpe.nimble -- Community package requirements: sanity tests hpe-storage/nimble-ansible-modules#63
  25 validate-modules
  23 compile-python-2.7
• infinidat.infinibox -- Community package requirements: sanity tests Infinidat/ansible-infinidat-collection#19
  24 validate-modules
  3 compile-python-3.10
  3 compile-python-3.11
  3 compile-python-3.12
  1 compile-python-2.7
• kubernetes.core -- Community package requirements: sanity tests ansible-collections/kubernetes.core#672
  25 validate-modules
  9 yamllint
• cisco.ise -- Community package requirements: sanity tests CiscoISE/ansible-ise#117
  15 validate-modules
  7 ansible-doc
  1 yamllint
• dellemc.unity -- Community package requirements: sanity tests dell/ansible-unity#47
  18 validate-modules
  3 compile-python-2.7
• netapp.azure -- Community package requirements: sanity tests ansible-collections/netapp.azure#9 (removed)
  16 validate-modules
• gluster.gluster -- Community package requirements: sanity tests gluster/gluster-ansible-collection#31
  10 validate-modules
• community.sap_libs -- Community package requirements: sanity tests sap-linuxlab/community.sap_libs#38
  9 validate-modules
• sensu.sensu_go -- Removal from Ansible (Community package requirements: sanity tests and repository management) sensu/sensu-go-ansible#362
  7 validate-modules
  1 WRONG_HASH
• community.okd -- Community package requirements: sanity tests openshift/community.okd#219
  7 validate-modules
• openstack.cloud -- https://bugs.launchpad.net/ansible-collections-openstack/+bug/2050854
  5 compile-python-2.7
  2 yamllint
• community.digitalocean -- Community package requirements: sanity tests ansible-collections/community.digitalocean#347
  4 validate-modules
  2 compile-python-2.7
• purestorage.fusion -- Community package requirements: sanity tests Pure-Storage-Ansible/Fusion-Collection#166
  6 compile-python-2.7
• netapp.um_info -- Community package requirements: sanity tests ansible-collections/netapp.um_info#11 (removed)
  5 validate-modules
• community.general -- Community package requirements: sanity tests ansible-collections/community.general#7884
  4 BANNED_SANITY
• community.mysql -- Community package requirements: sanity tests ansible-collections/community.mysql#608
  4 BANNED_SANITY
• netapp.aws -- Community package requirements: sanity tests ansible-collections/netapp.aws#9 (removed)
  4 validate-modules
• ansible.posix -- Community package requirements: sanity tests ansible-collections/ansible.posix#528
  3 BANNED_SANITY
• chocolatey.chocolatey -- Fix documentation errors in win_chocolatey and win_chocolatey_facts chocolatey/chocolatey-ansible#144
  3 validate-modules
• cyberark.conjur -- Removal from Ansible (Community package requirements: sanity tests) cyberark/ansible-conjur-collection#201
  2 yamllint
  1 validate-modules
• ibm.spectrum_virtualize (DEPRECATED)
  3 WRONG_HASH
• awx.awx
  1 BANNED_SANITY
  1 WRONG_HASH
• cisco.mso -- Community package requirements: sanity tests (DCNE-250) CiscoDevNet/ansible-mso#572
  2 validate-modules
• community.mongodb
  2 yamllint
• community.sap -- Community package requirements: sanity tests ansible-collections/community.sap#32
  2 yamllint
• lowlydba.sqlserver
  2 WRONG_HASH
• netbox.netbox
  2 compile-python-2.7
• vmware.vmware_rest
  2 validate-modules
• cisco.asa
  1 compile-python-2.7
• cisco.dnac -- Community package requirements: sanity tests cisco-en-programmability/dnacenter-ansible#208
  1 compile-python-2.7
• community.postgresql -- Fixed postgresql_tablespace: Fix sanity check requirements violation ansible-collections/community.postgresql#664
  1 BANNED_SANITY
• community.zabbix
  1 compile-python-2.7
• containers.podman
  1 compile-python-2.7
• netapp.storagegrid
  1 validate-modules
• purestorage.flasharray
  1 compile-python-2.7
• cyberark.pas -- Community package requirements: sanity tests and repository management cyberark/ansible-security-automation-collection#74
  2 WRONG_HASH
  1 validate-modules
• netapp.ontap — Community package requirements: sanity tests ansible-collections/netapp.ontap#224
• cisco.aci -- Community package requirements: sanity tests (DCNE-262) CiscoDevNet/ansible-aci#710

[mariolenz]

I think you can ignore some of those collections since they'll be removed from Ansible 10 anyway:

• Remove netapp.aws from Ansible 10 ansible-build-data#243
• Remove gluster.gluster from Ansible 10 ansible-build-data#249
• Remove community.sap from Ansible 10 ansible-build-data#266
• Remove netapp.azure from Ansible 10 ansible-build-data#272
• Remove netapp.um_info from Ansible 10 ansible-build-data#280
• Remove hpe.nimble from Ansible 10 ansible-build-data#285

[mariolenz]

BTW I see purestorage.fusion on your list. It looks like this collection will be deprecated. So maybe you can ignore this, too.

[mariolenz]

BTW I see purestorage.fusion on your list. It looks like this collection will be deprecated. So maybe you can ignore this, too.

@gotmax23 This collection has been deprecated by it's maintainers and will be removed from Ansible 10: ansible-community/ansible-build-data#374

[gotmax23]

Results for 10.3.0 are listed below and also represented in https://github.com/ansible-community/package-test-results/tree/main/rendered/10.3.0. Some of the duplicates from above may have been fixed in git but never released. Other have not fixed all the issues or not responded at all.

(Duplicates that are already listed in the issue description and that have had issues filled already should be checked off here.)

• cisco.intersight
  15582 validate-modules
• cisco.ucs
  9502 validate-modules
• ovirt.ovirt
  136 WRONG_HASH
  113 yamllint
  66 validate-modules
  60 compile-python-3.10
  60 compile-python-3.11
  60 compile-python-3.12
  59 compile-python-3.7
  59 compile-python-3.8
  59 compile-python-3.9
  11 ansible-doc
  1 WRONG_GALAXY_YML_NAME
• community.network
  654 BANNED_SANITY
• netapp_eseries.santricity
  342 validate-modules
  3 ansible-doc
  2 yamllint
• check_point.mgmt
  261 validate-modules
  4 WRONG_HASH
• google.cloud
  186 validate-modules
• theforeman.foreman
  122 yamllint
• wti.remote
  85 validate-modules
  7 ansible-doc
  4 yamllint
• ansible.posix
  9 validate-modules
• sensu.sensu_go
  7 validate-modules
  1 WRONG_HASH
• netapp.ontap
  4 validate-modules
• chocolatey.chocolatey
  3 validate-modules
• cyberark.conjur
  2 yamllint
  1 validate-modules
• cyberark.pas
  2 WRONG_HASH
  1 validate-modules
• ibm.spectrum_virtualize
  3 WRONG_HASH
• kubernetes.core
  3 BANNED_SANITY
• awx.awx
  1 BANNED_SANITY
  1 WRONG_HASH
• community.mongodb
  2 yamllint
• community.okd
  2 BANNED_SANITY
• ibm.storage_virtualize
  2 WRONG_HASH
• inspur.ispim
  2 WRONG_HASH
• kubevirt.core
  2 WRONG_HASH
• openstack.cloud
  2 yamllint

[gotmax23]

Followed up with:

• Community package requirements: sanity tests and CI wtinetworkgear/wti-collection#15
• Removal from Ansible (Community package requirements: sanity tests) cyberark/ansible-conjur-collection#201 — https://forum.ansible.com/t/collection-removal-cyberark-conjur/8381/1
• Removal from Ansible (Community package requirements: sanity tests and repository management) sensu/sensu-go-ansible#362 — https://forum.ansible.com/t/collection-removal-sensu-sensu-go/8380
• (Removal from Ansible) Community package requirements: sanity tests ansible-collections/google.cloud#613 — https://forum.ansible.com/t/8609

(Will follow up again in a couple weeks)

[gotmax23]

@gundalow followed up with

• Community package requirements: sanity tests NetApp/santricity#23

(We should follow up again in a couple weeks)

[gotmax23]

• Community package requirements: sanity tests CiscoDevNet/ansible-ucs#40 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.ucs.md)
• Community package requirements: sanity tests CiscoDevNet/intersight-ansible#108 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.intersight.md)

are also a problem. The maintainer says that our results are broken and differ from the results reported by Automation Hub. Perhaps there are sanity ignore entries that are not included in the git repository. Otherwise, I am at a loss.

[gotmax23]

@gundalow, any idea what's going on with the Cisco collections? Looking at the output in https://console.redhat.com/ansible/automation-hub/repo/published/cisco/intersight/import-log/, it seems that AH doesn't run the validate-modules test which is one that we require to pass.

[gotmax23]

A discussion about community.network has been started in https://forum.ansible.com/t/should-we-remove-community-network-from-the-ansible-community-package/8030.

[felixfontein]

• Community package requirements: sanity tests CiscoDevNet/ansible-ucs#40 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.ucs.md)

  • Community package requirements: sanity tests CiscoDevNet/intersight-ansible#108 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.intersight.md)

are also a problem. The maintainer says that our results are broken and differ from the results reported by Automation Hub. Perhaps there are sanity ignore entries that are not included in the git repository. Otherwise, I am at a loss.

It turned out that this is actually due to a bug in ansible-test (ansible/ansible#83842). On the other hand, if Automation Hub's sanity tests accepted it, it means that AH doesn't run the validate-modules sanity test. This basically means that successful import in AH doesn't mean that all sanity tests pass. (But it also doesn't mean that the ones not run fail...)

[gotmax23]

10.4.0

(Duplicates that are already listed in the issue description and that have had issues filled already should be checked off here.)

• ovirt.ovirt
  136 WRONG_HASH
  113 yamllint
  66 validate-modules
  60 compile-python-3.10
  60 compile-python-3.11
  60 compile-python-3.12
  59 compile-python-3.7
  59 compile-python-3.8
  59 compile-python-3.9
  11 ansible-doc
  1 WRONG_GALAXY_YML_NAME
• community.network
  654 BANNED_SANITY
• netapp_eseries.santricity
  342 validate-modules
  3 ansible-doc
  2 yamllint
• check_point.mgmt
  261 validate-modules
  4 WRONG_HASH
• google.cloud
  203 validate-modules
• theforeman.foreman
  122 yamllint
• wti.remote
  85 validate-modules
  4 WRONG_HASH
• ansible.posix
  9 validate-modules
• sensu.sensu_go
  7 validate-modules
  1 WRONG_HASH
• dellemc.openmanage
  4 WRONG_HASH
• netapp.ontap
  4 validate-modules
• chocolatey.chocolatey
  3 validate-modules
• cyberark.conjur
  2 yamllint
  1 validate-modules
• cyberark.pas
  2 WRONG_HASH
  1 validate-modules
• ibm.spectrum_virtualize
  3 WRONG_HASH
• kubernetes.core
  3 BANNED_SANITY
• awx.awx
  1 BANNED_SANITY
  1 WRONG_HASH
• community.mongodb
  2 yamllint
• community.okd
  2 BANNED_SANITY
• ibm.storage_virtualize
  2 WRONG_HASH
• inspur.ispim
  2 WRONG_HASH
• kubevirt.core
  2 WRONG_HASH
• openstack.cloud
  2 yamllint
• cisco.intersight
  1 WRONG_HASH
• dellemc.enterprise_sonic
  1 WRONG_HASH

Total: 25 collections

[mariolenz]

@gotmax23 You've started and named this issue [TRACKER] 9.1.0 collection test failures. But in your last comment you mention 10.4.0. I'm a little bit confused...

I suggest to open an new [TRACKER] 10.x collection test failures (and a [TRACKER] 11.x collection test failures once it's been released) issue. And close this one once Ansible 9.x is EOL. I think this would make it easier for us to keep track.

What do you think?