feat(scripts): init set tags (#188)

* ✨ scripts init set_tags

* 📝 scripts init README

* docs(scripts): update set_tags help and README

* docs(scripts): README update quotes escaping

* feat(scripts): set_tags implement suggested error handling

* chore(scripts): set_tags cleanup enumeration

* docs(scripts): README set config lang

Author: simonhir

README.md

@@ -16,6 +16,7 @@ Following a list and short description of the different components:
 - [dispatch-service](./dispatch-service): Service for notifying other services (i.e. DMS) that a file is ready for further processing via Kafka.
 - [handler-core](./handler-core): Library with base components for building a new service for handling notifications from the `dispatch-service`.
 - [dms-service](./dms-service): Service for transferring files into DMS when notified by the `dispatch-service` via Kafka. Based on `handler-core`.
+- [scripts](./scripts): Helper scripts for maintenance tasks.
 
 ## Contributing
 

scripts/README.md

@@ -0,0 +1,33 @@
+# Scripts
+
+Helper scripts for SWIM maintenance tasks.
+
+## Installation
+
+- Requires Python 3
+- Install dependencies with `pip install -r requirements.txt`
+- Configure custom CA in `~/.aws/config` (if required)
+```ini
+[default]
+ca_bundle = ~/root_cert.pem
+```
+- (Optional) Configure profiles in `~/.aws/credentials` (Ensure security)
+```ini
+[local]
+aws_access_key_id = minio
+aws_secret_access_key = Test1234
+endpoint_url = http://localhost:9000
+```
+
+## Execute
+
+See the command help for a description of the different arguments.
+
+```sh
+# help
+python ./set_tags.py --help
+# example with arguments (asks for access-key and secret-key if not defined)
+python ./set_tags.py --endpoint-url http://localhost:9000 --bucket swim-bucket --prefix test-meta/inProcess/ --tagging '{\"SWIM_State\": \"processed\"}'
+# example with profile (endpoint-url, access-key and secret-key need to be defined in profile)
+python ./set_tags.py --profile local --bucket swim-bucket --prefix test-meta/inProcess/ --tagging '{\"SWIM_State\": \"processed\"}'
+```

scripts/requirements.txt

@@ -0,0 +1 @@
+boto3==1.37.38

scripts/set_tags.py

@@ -0,0 +1,74 @@
+import argparse
+import getpass
+import json
+from typing import Optional, Dict
+
+import boto3
+from botocore.client import BaseClient
+
+
+def get_s3_client(profile: Optional[str], endpoint_url: Optional[str], access_key: Optional[str], secret_key: Optional[str]):
+    if profile:
+        session = boto3.Session(profile_name=profile)
+        return session.client("s3")
+    elif endpoint_url and access_key and secret_key:
+        return boto3.client("s3", endpoint_url=endpoint_url, aws_access_key_id=access_key, aws_secret_access_key=secret_key)
+    else:
+        raise ValueError("You must provide either a profile or endpoint URL and access/secret key.")
+
+def set_tags(s3, bucket: str, key: str, tags: Dict[str, str]) -> bool:
+    tag_set = [{'Key': k, 'Value': v} for k, v in tags.items()]
+    try:
+        s3.put_object_tagging(Bucket=bucket, Key=key, Tagging={'TagSet': tag_set})
+        return True
+    except Exception as e:
+        print(f"Error tagging object {key}: {e}")
+        return False
+
+def tag_files_recursive(s3: BaseClient, bucket: str, key: str, tags: Dict[str, str]):
+    paginator = s3.get_paginator('list_objects_v2')
+    page_iterator = paginator.paginate(Bucket=bucket, Prefix=key)
+    total_tagged = 0
+    for page_number, page in enumerate(page_iterator):
+        objects = [obj['Key'] for obj in page.get('Contents', []) if 'Size' in obj]
+        print(f"Page {page_number + 1}, Size: {len(objects)}")
+        for key in objects:
+            if set_tags(s3, bucket, key, tags):
+                total_tagged += 1
+    return total_tagged
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Tag objects in S3 bucket"
+    )
+
+    parser.add_argument("--profile", type=str, help="AWS profile name")
+    parser.add_argument("--endpoint-url", type=str, help="AWS endpoint URL")
+    parser.add_argument("--access-key", type=str, help="AWS access key")
+    parser.add_argument("--secret-key", type=str, help="AWS secret key")
+    parser.add_argument("--bucket", type=str, required=True, help="S3 bucket name")
+    parser.add_argument("--prefix", type=str, required=True, help="prefix to tag objects under. E.g. foo/bar/")
+    parser.add_argument("--tagging", type=str, required=True, help="tags in format of flat JSON map. E.g. '{\"key\":\"value\"}'")
+
+    args = parser.parse_args()
+
+    # Ask for endpoint URL, access and secret key if not defined nor profile
+    if not args.profile:
+        if not args.endpoint_url:
+            args.endpoint_url = input("Enter endpoint URL: ")
+        if not args.access_key:
+            args.access_key = input("Enter access key: ")
+        if not args.secret_key:
+            args.secret_key = getpass.getpass("Enter secret key: ")
+
+    # Parse tags
+    tags = json.loads(args.tagging)
+
+    # Tag objects
+    print("Starting tagging...")
+    s3 = get_s3_client(profile=args.profile, endpoint_url=args.endpoint_url, access_key=args.access_key, secret_key=args.secret_key)
+    tagged_files = tag_files_recursive(s3=s3, bucket=args.bucket, key=args.prefix, tags=tags)
+    print(f"Tagged {tagged_files} files")
+
+if __name__ == "__main__":
+    main()