build issue - createVirtualEnvironment -> pygradleBootstrap

[mjothiram]

I see this issue in my build from this morning. And trying to see this PyPi in LinkedIn Artifactory, but could not see that. Is it removed or any issue?

Thanks

Execution failed for task ':airflow:createVirtualEnvironment'.

Could not resolve all files for configuration ':airflow:pygradleBootstrap'.
Could not resolve pypi:virtualenv:16.0.0.
Required by:
project :airflow
> Could not resolve pypi:virtualenv:16.0.0.
> Could not get resource 'https://linkedin.jfrog.io/linkedin/pypi-external/pypi/virtualenv/16.0.0/virtualenv-16.0.0.ivy'.
> Could not GET 'https://linkedin.jfrog.io/linkedin/pypi-external/pypi/virtualenv/16.0.0/virtualenv-16.0.0.ivy'. Received status code 401 from server: Unauthorized

[rafishaiks03]

Hi

Can anyone please address the above issue, iam also facing the same.

I'm not able to get a solution....

Thanks in Advance..!!

[sudhir418]

Even we are facing the same issue , any alternatives?

[byblakeorriver]

Same here.

[rafishaiks03]

I'm still facing the same issue and didn't find any alternatives also..
still looking for help.. if not dropping Pygradle is the only option left...

[griffinptg]

I was able to resolve by downloading dependencies from pypi and uploading to my company's Artifactory. I had to make an .ivy file for each. It was a pain!

For example:
Download the .tar.gz here: https://pypi.org/project/virtualenv/16.0.0/
Upload that to your own Artifactory (assuming you have one) with an ivy file, virtualenv-16.0.0.ivy:

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="pypi" module="virtualenv" revision="16.0.0" />
  <configurations>
    <conf name="default" description="auto generated configuration for default" />
    <conf name="source" description="auto generated configuration for source" extends="default" />
  </configurations>
  <publications>
    <artifact name="virtualenv" ext="tar.gz" conf="default" type="tar.gz" />
  </publications>
  <dependencies defaultconfmapping="*-&gt;default" />
</ivy-module>

There are a bunch more after this FYI. And if you get a missing Python import instead of the 401, do the same, but also add it as a dependency to the lib that needs it, for example:

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="pypi" module="requests" revision="2.18.4" />
  <configurations>
    <conf name="default" description="auto generated configuration for default" />
    <conf name="source" description="auto generated configuration for source" extends="default" />
  </configurations>
  <publications>
    <artifact name="requests" ext="tar.gz" conf="default" type="tar.gz" />
  </publications>
  <dependencies defaultconfmapping="*-&gt;default">
    <dependency org="pypi" name="urllib3" rev="1.26.6" conf="default" />
    <dependency org="pypi" name="chardet" rev="4.0.0" conf="default" />
    <dependency org="pypi" name="certifi" rev="2021.5.30" conf="default" />
    <dependency org="pypi" name="idna" rev="3.2" conf="default" />
  </dependencies>
</ivy-module>

Just posting this as a workaround. It would still be good to see a fix for this.

[rafishaiks03]

Hi @griffinmm,

It seems you have posted in maven dependency format could you please help me out how and where to add the above code in gradle build. May be you can share your build.gradle file with the above things.

Thanks in Advance...!

[griffinptg]

@rafishaiks03 Here's my build.gradle, sanitized a bit:

buildscript {
    repositories {
        ivy {
            url 'http://artifactory.yourdomain.com/artifactory/plugins-release'
            credentials {
                username = "${artifactory_user}"
                password = "${artifactory_password}"
            }
        }
    }
    dependencies {
        classpath "org.jfrog.buildinfo:build-info-extractor-gradle:4.15.1"
    }
}

plugins {
    id 'com.linkedin.python' version '0.9.11'
}

allprojects {
    apply plugin: "com.jfrog.artifactory"
}

artifactory {
    contextUrl = "https://artifactory.yourdomain.com/artifactory"
    resolve {
        repository {
            repoKey = 'pypi-local'
            username = "${artifactory_user}"
            password = "${artifactory_password}"
            maven = false
            ivy {
                ivyLayout =  "[organisation]/[module]/[revision]/[module]-[revision].ivy"  
                artifactLayout =  "[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"
                mavenCompatible = false
            }
        }
    }
}

python {
   testDir = project.file('integration-test').path
   
   pythonEnvironment = loadEnvFile()

   details {
       pythonVersion = '3.6'
   }

   forceVersion('pypi', 'flake8', '3.2.1')
}

repositories {
   pyGradlePyPi()
}

pytest {
  additionalArguments = ["--junitxml=test-results/results.xml"]
}

dependencies {
  setupRequires 'pypi:jedi:0.11.1'
  setupRequires 'pypi:autopep8:1.3.3'
  setupRequires 'pypi:rope:0.10.7'
  setupRequires 'pypi:yapf:0.20.0'

  test 'pypi:requests:2.18.4'
  test('pypi:boto3:1.5.18') {
    exclude module: 'docutils'
  }
}

def loadEnvFile() {
  def m = [:]
  if(project.hasProperty('configFile')) {
    file('integration-test/config/' + configFile).eachLine { line ->
      def tokens = line.tokenize('=')
      println "Setting property: " + tokens
      m.put(tokens[0], tokens[1])
    }
  }
  return m
}

[alghoshal]

For anyone else facing these pygradle issue, been able to build on Python-3.8. Like @griffinmm had to download modules from pypi, with some differences:

• Downloaded them using the pivy-importer (see: https://github.com/linkedin/pygradle.git), which also prepares the ivy files automatically
• Downloaded modules to a local folder (specified in build.gradle) & used that as a local repository in place of Artifactory.
• Had to fix issues with name, case, dependencies, etc in some of modules (see Script-2 below: fixDownloadedIvyModules.sh)

Details shared here:
Project: https://github.com/alghoshal/pygradle_python3_example/blob/main/README.md

Build file: https://github.com/alghoshal/pygradle_python3_example/blob/main/build.gradle

Script-1 (Downloader): https://github.com/alghoshal/pygradle_python3_example/blob/main/scripts/downloadPyGradleDependecies.sh

Script-2 (Fixes name, etc): https://github.com/alghoshal/pygradle_python3_example/blob/main/scripts/fixDownloadedIvyModules.sh