`create_config` fails with multiple issues during deployment

[wickgit]

I'm trying to deploy ClusterFuzz to GCP following the production setup documentation and butler.py create_config just dies when it tries to enable the Cloud Source Repositories API (sourcerepo.googleapis.com).

Environment

• ClusterFuzz Version: master (commit 10fc2d92)
• Python Version: 3.11
• OS: Ubuntu 20.04
• GCP Project: Organization account

Command to run

python butler.py create_config \
  --oauth-client-secrets-path=/path/to/credentials.json \
  --firebase-api-key=<FIREBASE_KEY> \
  --project-id=<PROJECT_ID> \
  /path/to/config-dir

Expected Behavior

The script should either:

1. Skip optional APIs that cannot be enabled, OR
2. Provide clear documentation about which APIs are truly required vs optional, OR
3. Allow users to specify which APIs to skip via command-line flag

Actual Behavior

The script fails with a permission error and exits:

Running: gcloud --project=<PROJECT_ID> services enable pubsub.googleapis.com redis.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com secretmanager.googleapis.com siteverification.googleapis.com sourcerepo.googleapis.com stackdriver.googleapis.com storage-api.googleapis.com storage-component.googleapis.com vpcaccess.googleapis.com

ERROR: (gcloud.services.enable) [user@example.com] does not have permission to access projects instance [<PROJECT_ID>] (or it may not exist): Bind permission denied for service: sourcerepo.googleapis.com
Service sourcerepo.googleapis.com is not available to this consumer.
Help Token: <REDACTED>. This command is authenticated as user@example.com which is the active account specified by the [core/account] property
- '@type': type.googleapis.com/google.rpc.PreconditionFailure
  violations:
  - subject: '110002'
    type: googleapis.com
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: serviceusage.googleapis.com
  reason: AUTH_PERMISSION_DENIED
Return code is non-zero (1).

Traceback (most recent call last):
  File "/home/user/clusterfuzz/butler.py", line 459, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/clusterfuzz/butler.py", line 441, in main
    return command.execute(args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/clusterfuzz/src/local/butler/create_config.py", line 310, in execute
    enable_services(gcloud)
  File "/home/user/clusterfuzz/src/local/butler/create_config.py", line 153, in enable_services
    gcloud.run('services', 'enable', *_REQUIRED_SERVICES[i:i + end])
  File "/home/user/clusterfuzz/src/local/butler/common.py", line 54, in run
    return _run_and_handle_exception(arguments, GcloudError)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/clusterfuzz/src/local/butler/common.py", line 102, in _run_and_handle_exception
    raise exception_class(output)
local.butler.common.GcloudError: b"ERROR: (gcloud.services.enable) [user@example.com] does not have permission to access projects instance [<PROJECT_ID>] (or it may not exist): Bind permission denied for service: sourcerepo.googleapis.com\nService sourcerepo.googleapis.com is not available to this consumer.\nHelp Token: <REDACTED>. This command is authenticated as user@example.com which is the active account specified by the [core/account] property\n- '@type': type.googleapis.com/google.rpc.PreconditionFailure\n  violations:\n  - subject: '110002'\n    type: googleapis.com\n- '@type': type.googleapis.com/google.rpc.ErrorInfo\n  domain: serviceusage.googleapis.com\n  reason: AUTH_PERMISSION_DENIED\n"

Root Cause

Cloud Source Repositories API (sourcerepo.googleapis.com) is:

1. Included in _REQUIRED_SERVICES tuple in src/local/butler/create_config.py:60
2. Deprecated by Google

Workaround

Manually comment out the API in src/local/butler/create_config.py:

_REQUIRED_SERVICES = (
    # ... 
    'siteverification.googleapis.com',
    # 'sourcerepo.googleapis.com',
    'stackdriver.googleapis.com',
    # ... 
)

After commenting out the sourcerepo.googleapis.comanother error shows up:

Running: git rev-parse --short HEAD
| ea631dc1
Running: git -C /home/user1/clusterfuzz-config rev-parse --short HEAD
| fatal: not a git repository (or any of the parent directories): .git
| Return code is non-zero (128).
| Exit.
Traceback (most recent call last):
  File "/home/user/clusterfuzz/butler.py", line 459, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/clusterfuzz/butler.py", line 441, in main
    return command.execute(args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/clusterfuzz/src/local/butler/create_config.py", line 349, in execute
    deploy_terraform(args.new_config_dir)
  File "/home/user/clusterfuzz/src/local/butler/create_config.py", line 220, in deploy_terraform
    subprocess.check_call([
  File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python3', 'butler.py', 'deploy', '--force', '--targets', 'terraform', '--prod', '--config-dir', '/home/user/clusterfuzz-config']' returned non-zero exit status 128.

Config directory needs to be a git repo but this isn't documented anywhere. Even if you initialize git beforehand, it doesn't work because the script deletes and recreates the config directory (line 180-182 in create_config.py):

if os.path.exists(new_config_dir):
  print('Overwriting existing directory.')
  shutil.rmtree(new_config_dir)  # Deletes the config dir

So you can't pre-initialize git, and the script doesn't initialize it after creating the directory.

Proposed Solution

1. Remove sourcerepo.googleapis.com from _REQUIRED_SERVICES
2. Add error handling to continue if optional APIs fail to enable
3. Document which APIs are truly required vs optional
4. Either auto-initialize config directory as git repo OR add clear error message about this requirement

---

Is there any workaround to get create_config working without modifying the source code?

[wickgit]

@DavidKorczynski @oliverchang @hunsche