diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index fe5a208f..4d711a3c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -12,7 +12,7 @@ from math import sqrt def main(): sqrt(2) - + if __name__ == "__main__": main() ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index db43d22d..e109cdce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,11 +7,18 @@ - Expanded Training Workflows section in `docs/index.md` with 10 educational workflows including RGB/RYB color matching, titration, yeast growth optimization, vision-enabled 3D printing optimization, microscopy image stitching, and AprilTag robot path planning. - Research Workflows section in `docs/index.md` documenting alkaline catalysis lifecycle testing and battery slurry viscosity optimization. - Direct links from unit operations and workflows to relevant code locations in the repository for easier navigation. +- Resolution setting in `my_secrets_example.py` for YouTube-compatible streaming (144p, 240p, 360p, 480p, 720p, 1080p). +- Camera rotation setting (0, 90, 180, 270 degrees) for portrait mode streaming in `my_secrets_example.py`. +- Frame rate setting in `my_secrets_example.py` for adjustable stream frame rate. +- Timestamp overlay setting in `my_secrets_example.py` to display date/time (format: YYYY-MM-DD_HH-MM-SS) on video stream. ### Fixed - Ctrl+C interrupt handling in `src/ac_training_lab/picam/device.py` now properly exits the streaming loop instead of restarting. - Fixed typo "reagants" → "reagents" in Conductivity workflow description. +### Changed +- Removed CLI/argparse code from `device.py`; resolution and rotation are now configured via `my_secrets.py`. + ## [1.1.0] - 2024-06-11 ### Added - Imperial (10-32 thread) alternative design to SEM door automation bill of materials in `docs/sem-door-automation-components.md`. diff --git a/docs/sem-door-automation-components.md b/docs/sem-door-automation-components.md index 72f44eca..e88e99f1 100644 --- a/docs/sem-door-automation-components.md +++ b/docs/sem-door-automation-components.md @@ -2,23 +2,23 @@ ## 1. Adjustable Tie Rod (M4, Stainless Steel) -**Product:** 18-8 Stainless Steel Threaded Rod, M4 × 0.7 mm, 50 mm Length, Fully Threaded -**Link:** [McMaster-Carr 98863A534](https://www.mcmaster.com/98863A534/) -**Justification:** Provides corrosion resistance and compatibility with M4 rod ends and actuator clevis. 50 mm length is suitable for most SEM linkage applications and can be further cut to length if needed. Use with jam nuts for adjustment and secure installation. +**Product:** 18-8 Stainless Steel Threaded Rod, M4 × 0.7 mm, 50 mm Length, Fully Threaded +**Link:** [McMaster-Carr 98863A534](https://www.mcmaster.com/98863A534/) +**Justification:** Provides corrosion resistance and compatibility with M4 rod ends and actuator clevis. 50 mm length is suitable for most SEM linkage applications and can be further cut to length if needed. Use with jam nuts for adjustment and secure installation. ## 2. Ball Joint Rod End (M4, Stainless Steel) -**Product:** Internally Threaded Corrosion-Resistant Ball Joint Rod End, M4 × 0.7 mm Thread, 303 Stainless Steel -**Link:** [McMaster-Carr 2988K31](https://www.mcmaster.com/2988K31) +**Product:** Internally Threaded Corrosion-Resistant Ball Joint Rod End, M4 × 0.7 mm Thread, 303 Stainless Steel +**Link:** [McMaster-Carr 2988K31](https://www.mcmaster.com/2988K31) **Justification:** Stainless steel construction ensures corrosion resistance in laboratory environments. Ball joint allows for angular misalignment and smooth actuation. M4 thread is compatible with the tie rod and actuator clevis. ## 3. Jam Nut (M4, Stainless Steel) -**Product:** Stainless Steel Hex Jam Nut, M4 × 0.7 mm Thread -**Link:** [McMaster-Carr 91828A110](https://www.mcmaster.com/91828A110) +**Product:** Stainless Steel Hex Jam Nut, M4 × 0.7 mm Thread +**Link:** [McMaster-Carr 91828A110](https://www.mcmaster.com/91828A110) **Justification:** Secures the rod end to the tie rod, preventing loosening during repeated actuation. Stainless steel for durability and corrosion resistance. ## 4. Handle Attachment (Cable Tie or U-Bolt Clamp) -**Product:** Select based on SEM handle geometry. Recommended: Stainless steel cable tie or U-bolt clamp sized to fit handle diameter. -**Link:** [Cable Ties](https://www.mcmaster.com/cable-ties/) | [U-Bolt Clamps](https://www.mcmaster.com/u-bolts/) +**Product:** Select based on SEM handle geometry. Recommended: Stainless steel cable tie or U-bolt clamp sized to fit handle diameter. +**Link:** [Cable Ties](https://www.mcmaster.com/cable-ties/) | [U-Bolt Clamps](https://www.mcmaster.com/u-bolts/) **Justification:** Provides a secure, non-slip connection to the SEM handle. Stainless steel options preferred for longevity and chemical resistance. ### Cable Tie for SEM Handle Attachment diff --git a/docs/tailscale-setup.md b/docs/tailscale-setup.md index f331749e..e244d257 100644 --- a/docs/tailscale-setup.md +++ b/docs/tailscale-setup.md @@ -17,7 +17,7 @@ For running commands on your device during setup, you have two options: # Using mDNS/Bonjour hostname (if supported): ssh @.local # Example: ssh ac@rpi-zero2w-stream-cam-a1b2.local - + # Or using the device's local IP address: ssh @ # Replace with your Pi's username (default is `pi`) and (e.g., 192.168.1.100) with your device's actual IP address. @@ -81,7 +81,7 @@ Additional resources: Note that you can only use remote desktop on Windows 10/11 Pro or Windows 10/11 Enterprise, not on Windows 10/11 Home. ```{warning} -[Install Tailscale for Windows](https://tailscale.com/kb/1022/install-windows). +[Install Tailscale for Windows](https://tailscale.com/kb/1022/install-windows). We recommend using a private browser for the interactive login step if this is a non-personal device. You may need to copy the auto-opened URL to the private browser manually. Next, set up the "Remote Desktop Protocol" (RDP) [according to Tailscale's documentation](https://tailscale.com/kb/1095/secure-rdp-windows). @@ -324,4 +324,3 @@ Now, you can reboot OT-2 and see if the device on the admin page of Tailscale wi | `/data/tailscale_1.82.0_arm/tailscale` | Tailscale CLI | | `/data/start_tailscale.sh` | Startup script | | `/etc/systemd/system/tailscale-autostart.service` | Systemd autostart service definition | - diff --git a/scripts/playwright/sem-open-close/chat-history.md b/scripts/playwright/sem-open-close/chat-history.md index 0d9c8a09..f0a64b19 100644 --- a/scripts/playwright/sem-open-close/chat-history.md +++ b/scripts/playwright/sem-open-close/chat-history.md @@ -1,17 +1,17 @@ sgbaird: Use #playwright to carry out the design task in #file:sem-door-automation-design-prompt.md to search through mcmaster.com. You are also allowed to run Google searches or use parts from other sources, but the preference is for mcmaster.com. You may also use #perplexity_ask if you see fit. You are welcome to use #github to look up additional context from github issues. Actively update a new markdown file within the same folder. Justify the choices you make. Include direct hyperlinks. -GitHub Copilot: Ran Navigate to a URL +GitHub Copilot: Ran Navigate to a URL Completed with input: { "url": "https://www.mcmaster.com/" } -Ran Click +Ran Click Completed with input: { "element": "Search textbox at top of McMaster-Carr homepage", "ref": "e133" } -Ran Type text +Ran Type text Completed with input: { "element": "Search textbox at top of McMaster-Carr homepage", "ref": "e133", @@ -19,7 +19,7 @@ Completed with input: { "submit": true } -Ran Click +Ran Click Completed with input: { "element": "Connecting Rods category link (for adjustable tie rods)", "ref": "e4085" @@ -27,7 +27,7 @@ Completed with input: { Summarized conversation history -Ran Click +Ran Click Completed with input: { "element": "M4 thread size filter link", "ref": "e4238" @@ -35,15 +35,15 @@ Completed with input: { I have filtered McMaster-Carr's connecting rods to show only M4 thread size options. The next step is to select a compatible adjustable rod for your SEM door linkage, then document the product link and justification. I will also identify matching rod ends, jam nuts, and handle attachment solutions in the same way. -Proceeding to select and document the recommended components for your design. +Proceeding to select and document the recommended components for your design. -Ran Click +Ran Click Completed with input: { "element": "Internally Threaded Connecting Rods (for M4, 75mm, 18-8 Stainless Steel)", "ref": "e5095" } -Ran Click +Ran Click Completed with input: { "element": "right-hand threaded rod ends (for M4)", "ref": "e4954" @@ -58,7 +58,7 @@ I have identified the following recommended McMaster-Carr components for your SE **2. Rod End (Internally Threaded, M4, Stainless Steel)** - Next step: Filter for M4 internally threaded ball joint rod ends in stainless steel for optimal alignment and durability. -I will now document these selections and continue with jam nuts and handle attachment solutions. +I will now document these selections and continue with jam nuts and handle attachment solutions. Created [](file:///c%3A/Users/bairdst4/Documents/GitHub/AccelerationConsortium/ac-training-lab/docs/sem-door-automation-components.md) @@ -70,13 +70,13 @@ Made changes. sgbaird: Go ahead -GitHub Copilot: Ran Click +GitHub Copilot: Ran Click Completed with input: { "element": "Internally Threaded Ball Joint Rod Ends (for M4, Stainless Steel)", "ref": "e7036" } -Ran Click +Ran Click Completed with input: { "element": "M4 jam nut (stainless steel)", "ref": "e5572" @@ -98,15 +98,15 @@ The bill of materials and engineering notes for the SEM door automation linkage Made changes. -sgbaird: Provide a specific part for the cable ties based on the design requirements in #file:sem-door-automation-design-prompt.md +sgbaird: Provide a specific part for the cable ties based on the design requirements in #file:sem-door-automation-design-prompt.md -GitHub Copilot: Ran Click +GitHub Copilot: Ran Click Completed with input: { "element": "Stainless Steel Cable Ties, 200mm length, 4.6mm width (for 20-24mm handle diameter)", "ref": "e1001" } -Ran Click +Ran Click Completed with input: { "element": "Stainless Steel Cable Ties (select 200mm length, 4.6mm width for 20-24mm handle)", "ref": "e7221" @@ -120,7 +120,7 @@ Summarized conversation historyThe Playwright tool is currently focused on the r - **Direct Link:** [McMaster-Carr 7131K46](https://www.mcmaster.com/7131K46/) - **Engineering Justification:** This cable tie is corrosion-resistant, fits the 20–24 mm handle diameter, and provides secure, low-profile clamping for the SEM handle geometry. -I will update your bill of materials markdown file with this part number, link, and rationale. +I will update your bill of materials markdown file with this part number, link, and rationale. Using "Apply Patch" @@ -140,33 +140,33 @@ sgbaird: https://www.mcmaster.com/6058K42 looks like the incorrect link for the 7131K46 also looks like the incorrect link for a stainless steel cable tie. Use #playwright to open a new tab if needed, or perform a new search. Fetch and validate links/part numbers to make sure they match the descriptions you're giving. -GitHub Copilot: Ran Manage tabs +GitHub Copilot: Ran Manage tabs Completed with input: { "action": "new" } -Ran Navigate to a URL +Ran Navigate to a URL Completed with input: { "url": "https://www.mcmaster.com/adjustable-tie-rods/" } -Ran Navigate to a URL +Ran Navigate to a URL Completed with input: { "url": "https://www.mcmaster.com/stainless-steel-cable-ties/" } -Ran Get console messages +Ran Get console messages Completed with input: { "onlyErrors": true } -Ran Page snapshot +Ran Page snapshot Completed with input: {} -Ran Page snapshot +Ran Page snapshot Completed with input: {} -Ran Click +Ran Click Completed with input: { "element": "Cable Ties (Stainless Steel)", "ref": "e1201" @@ -185,12 +185,12 @@ Would you like to proceed with updating the documentation, or do you need valida sgbaird: Yes, validate the other too. -GitHub Copilot: Ran Navigate to a URL +GitHub Copilot: Ran Navigate to a URL Completed with input: { "url": "https://www.mcmaster.com/adjustable-tie-rods/" } -Ran Evaluate JavaScript +Ran Evaluate JavaScript Completed with input: { "element": "main product table or product list", "function": "() => { const tables = Array.from(document.querySelectorAll('table')); if (tables.length) { return tables.map(table => table.innerText).join('\\n\\n'); } const products = Array.from(document.querySelectorAll('[data-mcm-product]')); if (products.length) { return products.map(p => p.innerText).join('\\n\\n'); } return 'No product table or product list found.'; }" @@ -200,7 +200,7 @@ The adjustable tie rod you selected (Internally Threaded Connecting Rod, 18-8 St Both the cable tie (6898K381) and the adjustable tie rod (6058K42) selections are validated and correct for your SEM door automation linkage. -I will now update your documentation to reflect the validated cable tie part number and link. +I will now update your documentation to reflect the validated cable tie part number and link. Using "Apply Patch" @@ -221,7 +221,7 @@ Would you like me to select and validate a currently available M4 stainless stee sgbaird: Yes. Please fetch the webpage after finding the link to validate -GitHub Copilot: Ran Navigate to a URL +GitHub Copilot: Ran Navigate to a URL Completed with input: { "url": "https://www.mcmaster.com/connecting-rods/" } @@ -236,7 +236,7 @@ To ensure your documentation is accurate and reproducible, I recommend selecting sgbaird: Run a #playwright search again and find a suitable part -GitHub Copilot: Ran Click +GitHub Copilot: Ran Click Completed with input: { "element": "Metric Internally Threaded Connecting Rods", "ref": "e1952" @@ -244,16 +244,16 @@ Completed with input: { Summarized conversation historyThe McMaster-Carr connecting rods catalog does not currently list any M4 (4 mm) stainless steel internally threaded connecting rods. The available options are for larger diameters (e.g., 3/8", 1/4", etc.), and the original part number (6058K42) is not present or valid. -**Next step:** I will search the McMaster-Carr catalog for metric stainless steel rods or alternative linkage components (such as threaded rods or stud connectors) that can be adapted for M4 thread and SEM door automation. I will then update your bill of materials with a validated, currently available part number and direct link. +**Next step:** I will search the McMaster-Carr catalog for metric stainless steel rods or alternative linkage components (such as threaded rods or stud connectors) that can be adapted for M4 thread and SEM door automation. I will then update your bill of materials with a validated, currently available part number and direct link. -Proceeding to select and document a suitable alternative. +Proceeding to select and document a suitable alternative. -Ran Navigate to a URL +Ran Navigate to a URL Completed with input: { "url": "https://www.mcmaster.com/threaded-rods/" } -Ran Click +Ran Click Completed with input: { "element": "Metric Stainless Steel Threaded Rods", "ref": "e1" @@ -261,12 +261,12 @@ Completed with input: { A suitable alternative is available: McMaster-Carr offers an M4 × 0.7 mm fully threaded rod in 18-8 stainless steel. This matches the required thread size and material for corrosion resistance. -**Recommended part:** -- **Product:** 18-8 Stainless Steel Threaded Rod, M4 × 0.7 mm -- **Link:** [McMaster-Carr M4 × 0.7 mm Threaded Rod](https://www.mcmaster.com/98863A107) +**Recommended part:** +- **Product:** 18-8 Stainless Steel Threaded Rod, M4 × 0.7 mm +- **Link:** [McMaster-Carr M4 × 0.7 mm Threaded Rod](https://www.mcmaster.com/98863A107) - **Justification:** Provides corrosion resistance and compatibility with M4 rod ends and actuator clevis. Can be cut to length and used with jam nuts for adjustment. -I will update your bill of materials and documentation accordingly. +I will update your bill of materials and documentation accordingly. Using "Apply Patch" @@ -282,9 +282,9 @@ Made changes. sgbaird: "No matches were found for "98863A107."" -GitHub Copilot: The part number 98863A107 does not match any current McMaster-Carr product. I will search for a valid M4 × 0.7 mm stainless steel threaded rod or equivalent linkage component in the McMaster-Carr catalog and update your documentation with a correct, available part number and link. +GitHub Copilot: The part number 98863A107 does not match any current McMaster-Carr product. I will search for a valid M4 × 0.7 mm stainless steel threaded rod or equivalent linkage component in the McMaster-Carr catalog and update your documentation with a correct, available part number and link. -Ran Type text +Ran Type text Completed with input: { "element": "Search box", "ref": "e133", @@ -294,14 +294,14 @@ Completed with input: { Summarized conversation history -Ran Click +Ran Click Completed with input: { "element": "Threaded Rods section (19 products)", "ref": "e6989", "button": "left" } -Ran Click +Ran Click Completed with input: { "element": "M4 × 0.7 mm, 18-8 Stainless Steel, 50 mm length, Part Number 98863A534", "ref": "e7316", @@ -324,9 +324,9 @@ Made changes. sgbaird: Create a second alternative design that is allowed to use imperial units. (10-32) -GitHub Copilot: I will add a second alternative design to your documentation using imperial units (10-32 thread). This will include a bill of materials, direct McMaster-Carr links, and engineering justification for each component. +GitHub Copilot: I will add a second alternative design to your documentation using imperial units (10-32 thread). This will include a bill of materials, direct McMaster-Carr links, and engineering justification for each component. -Ran Type text +Ran Type text Completed with input: { "element": "Search box", "ref": "e133", @@ -334,7 +334,7 @@ Completed with input: { "submit": true } -Ran Click +Ran Click Completed with input: { "element": "Threaded Rods section (37 products)", "ref": "e8430", @@ -345,7 +345,7 @@ Using "Fetch Web Page" Fetched https://www.mcmaster.com/98804A106/ -Ran Type text +Ran Type text Completed with input: { "element": "Search box", "ref": "e133", @@ -353,7 +353,7 @@ Completed with input: { "submit": true } -Ran Type text +Ran Type text Completed with input: { "element": "Search box", "ref": "e133", @@ -385,4 +385,4 @@ The imperial (10-32 thread) alternative design has been added to sem-door-automa Let me know if you need further modifications or additional sourcing. -Made changes. \ No newline at end of file +Made changes. diff --git a/scripts/playwright/sem-open-close/sem-door-automation-design-prompt.md b/scripts/playwright/sem-open-close/sem-door-automation-design-prompt.md index 1c395ea8..28d7bfd8 100644 --- a/scripts/playwright/sem-open-close/sem-door-automation-design-prompt.md +++ b/scripts/playwright/sem-open-close/sem-door-automation-design-prompt.md @@ -16,7 +16,7 @@ Design a mechanical linkage system to connect an Actuonix L16 linear actuator to ### Handle Dimensions: - **Cross-sectional shape**: Ellipsoidal - **Diameter (left-right)**: 24mm -- **Diameter (top-bottom)**: 20mm +- **Diameter (top-bottom)**: 20mm - **Distance from right side to first handle**: 120mm - **3D printed component width**: 150mm across - **Optimal attachment point**: Center of handle for parallel force application @@ -40,7 +40,7 @@ Design a mechanical linkage system to connect an Actuonix L16 linear actuator to Implement a **tie rod linkage system** consisting of: 1. **Connecting Rod ("Tie Rod")**: Adjustable-length rod with threaded ends -2. **Jam Nut**: For locking rod at desired length +2. **Jam Nut**: For locking rod at desired length 3. **Clevis Rod End**: Attaches to actuator's clevis interface 4. **Rod End Bolt**: Connects to handle attachment mechanism 5. **Handle Attachment**: Secures rod end bolt to ellipsoidal handle @@ -84,4 +84,4 @@ Implement a **tie rod linkage system** consisting of: --- -This design task combines mechanical engineering principles with practical laboratory automation needs, requiring careful consideration of component selection, force analysis, and installation constraints in a precision instrument environment. \ No newline at end of file +This design task combines mechanical engineering principles with practical laboratory automation needs, requiring careful consideration of component selection, force analysis, and installation constraints in a precision instrument environment. diff --git a/src/ac_training_lab/a1_cam/README.md b/src/ac_training_lab/a1_cam/README.md index c061557e..50ab2250 100644 --- a/src/ac_training_lab/a1_cam/README.md +++ b/src/ac_training_lab/a1_cam/README.md @@ -35,7 +35,7 @@ Key considerations: - **Region**: Choose a region close to your devices for lower latency (e.g., `us-east-2`) - **Bucket name**: Must be globally unique (e.g., `rpi-zero2w-toolhead-camera`) - **Object Ownership**: ACLs disabled (recommended) -- **Block Public Access settings**: +- **Block Public Access settings**: - For enhanced security, keep "Block all public access" enabled (recommended) - If you need to access images from external systems, use IAM-based access controls or generate signed URLs rather than making the bucket public - Only uncheck public access if you fully understand the security implications and need publicly accessible URLs (as shown in the screenshot) @@ -150,7 +150,7 @@ Create AWS IAM credentials with S3 access permissions. Follow the official guide - Save these credentials securely in a password manager - Click **Done** -**Security best practices**: +**Security best practices**: - Revoke credentials immediately if compromised - Never commit credentials to version control diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_AUTH_SETUP.md b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_AUTH_SETUP.md index 6fb3d03f..6f8d2d71 100644 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_AUTH_SETUP.md +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_AUTH_SETUP.md @@ -40,7 +40,7 @@ mosquitto_passwd mosquitto_passwd admin_user **Recommended Users:** - **device_user**: For Raspberry Pi devices (read requests, write responses) -- **client_user**: For notebooks/orchestrators (write requests, read responses) +- **client_user**: For notebooks/orchestrators (write requests, read responses) - **admin_user**: For debugging (full access) ### 3. ACL File (`mosquitto_acl`) @@ -151,7 +151,7 @@ Updated test scripts are provided in `_scripts/`: **Client:** ``` ✓ Client authenticated and connected - Username: client_user + Username: client_user ✓ Publish successful ✓ SUCCESS! Received response: Image URI: https://test-bucket.s3... @@ -221,7 +221,7 @@ client = mqtt.Client(client_id="") 3. **Minimum permissions**: Grant only necessary topic access 4. **Audit logs**: Monitor connection attempts and publish/subscribe patterns 5. **Separate credentials**: Different users for devices vs clients vs admins -6. **File permissions**: +6. **File permissions**: ```bash chmod 600 mosquitto_passwd # Only mosquitto user can read chmod 644 mosquitto_acl # World-readable is OK (no secrets) @@ -243,7 +243,7 @@ User: device_user Publish: rpi-zero2w/still-camera/${clientid}/response Subscribe: rpi-zero2w/still-camera/${clientid}/request -User: client_user +User: client_user Publish: rpi-zero2w/still-camera/+/request Subscribe: rpi-zero2w/still-camera/+/response ``` diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_TEST_RESULTS.md b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_TEST_RESULTS.md index d5ba2765..dee07d68 100644 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_TEST_RESULTS.md +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/MQTT_TEST_RESULTS.md @@ -7,7 +7,7 @@ Successfully validated the MQTT communication pattern using a local mosquitto br ### Test Environment - **Broker**: Mosquitto 2.0.18 (localhost:1883) - **Device Serial**: test-cam-01 -- **Topic Structure**: +- **Topic Structure**: - Request: `rpi-zero2w/still-camera/{DEVICE_SERIAL}/request` - Response: `rpi-zero2w/still-camera/{DEVICE_SERIAL}/response` diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/create_mosquitto_auth.py b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/create_mosquitto_auth.py index a79dcc7b..35960bc4 100755 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/create_mosquitto_auth.py +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/create_mosquitto_auth.py @@ -3,26 +3,28 @@ Create mosquitto authentication files for testing. Generates password and ACL files with proper format. """ -import hashlib import base64 +import hashlib import os + def generate_password_hash(password): """Generate mosquitto-compatible password hash (SHA512-PBKDF2)""" # For simplicity, we'll use basic SHA512 hash # Mosquitto also supports plaintext with $6$ prefix for SHA512 salt = os.urandom(12) - password_hash = hashlib.pbkdf2_hmac('sha512', password.encode(), salt, 101) - + password_hash = hashlib.pbkdf2_hmac("sha512", password.encode(), salt, 101) + # Mosquitto format: $6$salt$hash (base64 encoded) - salt_b64 = base64.b64encode(salt).decode('ascii') - hash_b64 = base64.b64encode(password_hash).decode('ascii') - + salt_b64 = base64.b64encode(salt).decode("ascii") + hash_b64 = base64.b64encode(password_hash).decode("ascii") + return f"$6${salt_b64}${hash_b64}" + # Create password file print("Creating /tmp/mosquitto_passwd...") -with open('/tmp/mosquitto_passwd', 'w') as f: +with open("/tmp/mosquitto_passwd", "w") as f: f.write(f"device_user:{generate_password_hash('device_password')}\n") f.write(f"client_user:{generate_password_hash('client_password')}\n") print("✓ Password file created") @@ -38,7 +40,7 @@ def generate_password_hash(password): topic read rpi-zero2w/still-camera/+/request topic write rpi-zero2w/still-camera/+/response -# Client user can write requests and read responses +# Client user can write requests and read responses user client_user topic write rpi-zero2w/still-camera/+/request topic read rpi-zero2w/still-camera/+/response @@ -48,13 +50,13 @@ def generate_password_hash(password): topic readwrite # """ -with open('/tmp/mosquitto_acl', 'w') as f: +with open("/tmp/mosquitto_acl", "w") as f: f.write(acl_content) print("✓ ACL file created") -print("\n" + "="*60) +print("\n" + "=" * 60) print("Authentication Configuration Created") -print("="*60) +print("=" * 60) print("\nUsers:") print(" device_user / device_password - Device credentials") print(" client_user / client_password - Client credentials") diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/mosquitto_acl b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/mosquitto_acl index 271a746a..e62bed17 100644 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/mosquitto_acl +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/mosquitto_acl @@ -7,7 +7,7 @@ user device_user topic read rpi-zero2w/still-camera/+/request topic write rpi-zero2w/still-camera/+/response -# Client user can write requests and read responses +# Client user can write requests and read responses user client_user topic write rpi-zero2w/still-camera/+/request topic read rpi-zero2w/still-camera/+/response diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_client.py b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_client.py index 8b0e73ce..e0cb2ea1 100755 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_client.py +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_client.py @@ -1,8 +1,9 @@ #!/usr/bin/env python3 """Test MQTT client with authentication""" import json -import time import sys +import time + import paho.mqtt.client as mqtt DEVICE_SERIAL = "test-cam-01" @@ -11,9 +12,10 @@ response_received = False + def on_connect(client, userdata, flags, rc): if rc == 0: - print(f"✓ Client connected successfully as 'client_user'") + print("✓ Client connected successfully as 'client_user'") client.subscribe(RESPONSE_TOPIC, qos=1) print(f"✓ Subscribed to: {RESPONSE_TOPIC}") else: @@ -21,6 +23,7 @@ def on_connect(client, userdata, flags, rc): if rc == 5: print(" Authentication failed - check username/password") + def on_message(client, userdata, msg): global response_received print(f"\n✓ Client received response on {msg.topic}") @@ -31,12 +34,13 @@ def on_message(client, userdata, msg): except Exception as e: print(f"✗ Error processing message: {e}") + client = mqtt.Client(client_id="test-client") client.username_pw_set("client_user", "client_pass") client.on_connect = on_connect client.on_message = on_message -print(f"Connecting to broker at localhost:1883 as 'client_user'...") +print("Connecting to broker at localhost:1883 as 'client_user'...") client.connect("127.0.0.1", 1883, 60) client.loop_start() @@ -44,10 +48,7 @@ def on_message(client, userdata, msg): time.sleep(1) # Send capture request -request = { - "command": "capture", - "timestamp": time.time() -} +request = {"command": "capture", "timestamp": time.time()} print(f"\n✓ Client publishing request to: {REQUEST_TOPIC}") print(f" Request: {request}") diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_device.py b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_device.py index be5e36bc..6770f835 100755 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_device.py +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_auth_device.py @@ -2,15 +2,17 @@ """Test MQTT device with authentication""" import json import time + import paho.mqtt.client as mqtt DEVICE_SERIAL = "test-cam-01" REQUEST_TOPIC = f"rpi-zero2w/still-camera/{DEVICE_SERIAL}/request" RESPONSE_TOPIC = f"rpi-zero2w/still-camera/{DEVICE_SERIAL}/response" + def on_connect(client, userdata, flags, rc): if rc == 0: - print(f"✓ Device connected successfully as 'device_user'") + print("✓ Device connected successfully as 'device_user'") client.subscribe(REQUEST_TOPIC, qos=1) print(f"✓ Subscribed to: {REQUEST_TOPIC}") else: @@ -18,32 +20,34 @@ def on_connect(client, userdata, flags, rc): if rc == 5: print(" Authentication failed - check username/password") + def on_message(client, userdata, msg): print(f"\n✓ Device received request on {msg.topic}") try: request = json.loads(msg.payload.decode()) print(f" Request: {request}") - + # Send response response = { "status": "success", "timestamp": time.time(), "message": "Image captured", - "s3_uri": f"s3://test-bucket/images/test-{int(time.time())}.jpg" + "s3_uri": f"s3://test-bucket/images/test-{int(time.time())}.jpg", } - + client.publish(RESPONSE_TOPIC, json.dumps(response), qos=1) print(f"✓ Device published response to: {RESPONSE_TOPIC}") print(f" Response: {response}") except Exception as e: print(f"✗ Error processing message: {e}") + client = mqtt.Client(client_id="test-device") client.username_pw_set("device_user", "device_pass") client.on_connect = on_connect client.on_message = on_message -print(f"Connecting to broker at localhost:1883 as 'device_user'...") +print("Connecting to broker at localhost:1883 as 'device_user'...") client.connect("127.0.0.1", 1883, 60) print("Device is ready and waiting for requests...") diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client.py b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client.py index a1d88fda..2a4bd8d0 100755 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client.py +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client.py @@ -4,9 +4,10 @@ This simulates the notebook/orchestrator behavior. """ import json -import paho.mqtt.client as mqtt -from queue import Queue, Empty import time +from queue import Empty, Queue + +import paho.mqtt.client as mqtt # Configuration MQTT_HOST = "localhost" @@ -17,18 +18,20 @@ data_queue = Queue() + def on_connect(client, userdata, flags, rc, properties=None): if rc == 0: - print(f"✓ Client connected to MQTT broker") + print("✓ Client connected to MQTT broker") print(f" Subscribing to: {CAMERA_WRITE_TOPIC}") client.subscribe(CAMERA_WRITE_TOPIC, qos=1) else: print(f"✗ Connection failed with code {rc}") + def on_message(client, userdata, msg): print(f"\n✓ Received response on topic: {msg.topic}") print(f" Payload: {msg.payload.decode()}") - + try: data = json.loads(msg.payload) print(f" Parsed data: {data}") @@ -37,59 +40,62 @@ def on_message(client, userdata, msg): except json.JSONDecodeError as e: print(f" ✗ JSON decode error: {e}") + def main(): - print("="*60) + print("=" * 60) print("Test Camera Client Starting") - print("="*60) + print("=" * 60) print(f"MQTT Host: {MQTT_HOST}:{MQTT_PORT}") print(f"Device Serial: {DEVICE_SERIAL}") print(f"Request Topic: {CAMERA_READ_TOPIC}") print(f"Response Topic: {CAMERA_WRITE_TOPIC}") print() - + client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="test-client") client.on_connect = on_connect client.on_message = on_message - + try: client.connect(MQTT_HOST, MQTT_PORT, keepalive=60) client.loop_start() - + # Give it a moment to connect time.sleep(1) - + # Send capture command msg = {"command": "capture_image"} payload = json.dumps(msg) - print(f"Sending capture command...") + print("Sending capture command...") print(f" Topic: {CAMERA_READ_TOPIC}") print(f" Payload: {payload}") - + result = client.publish(CAMERA_READ_TOPIC, payload, qos=1) print(f" Publish result: {result.rc} (0=success)") - + # Wait for response - print(f"\nWaiting for response (timeout: 10s)...") + print("\nWaiting for response (timeout: 10s)...") try: data = data_queue.get(True, 10) - print(f"\n✓ SUCCESS! Received response:") + print("\n✓ SUCCESS! Received response:") print(f" Image URI: {data.get('image_uri')}") print(f" Timestamp: {data.get('timestamp')}") print(f" Device: {data.get('device_serial')}") except Empty: - print(f"\n✗ TIMEOUT: No response received after 10s") - print(f" Check that device is running and topics match") - + print("\n✗ TIMEOUT: No response received after 10s") + print(" Check that device is running and topics match") + except KeyboardInterrupt: print("\n\nShutting down...") except Exception as e: print(f"✗ Error: {e}") import traceback + traceback.print_exc() finally: client.loop_stop() client.disconnect() print("\nClient disconnected") + if __name__ == "__main__": main() diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client_auth.py b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client_auth.py index 063c2da9..4e4b1bad 100755 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client_auth.py +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_client_auth.py @@ -4,9 +4,10 @@ Tests username/password authentication and topic ACL filters. """ import json -import paho.mqtt.client as mqtt -from queue import Queue, Empty import time +from queue import Empty, Queue + +import paho.mqtt.client as mqtt # Configuration MQTT_HOST = "localhost" @@ -19,9 +20,10 @@ data_queue = Queue() + def on_connect(client, userdata, flags, rc, properties=None): if rc == 0: - print(f"✓ Client authenticated and connected") + print("✓ Client authenticated and connected") print(f" Username: {MQTT_USERNAME}") print(f" Subscribing to: {CAMERA_WRITE_TOPIC}") result = client.subscribe(CAMERA_WRITE_TOPIC, qos=1) @@ -32,17 +34,19 @@ def on_connect(client, userdata, flags, rc, properties=None): 2: "Connection refused - invalid client identifier", 3: "Connection refused - server unavailable", 4: "Connection refused - bad username or password", - 5: "Connection refused - not authorized" + 5: "Connection refused - not authorized", } print(f"✗ Connection failed: {error_messages.get(rc, f'Unknown error ({rc})')}") + def on_subscribe(client, userdata, mid, granted_qos, properties=None): print(f"✓ Subscription confirmed (QoS: {granted_qos})") + def on_message(client, userdata, msg): print(f"\n✓ Received response on topic: {msg.topic}") print(f" Payload: {msg.payload.decode()}") - + try: data = json.loads(msg.payload) print(f" Parsed data: {data}") @@ -51,69 +55,72 @@ def on_message(client, userdata, msg): except json.JSONDecodeError as e: print(f" ✗ JSON decode error: {e}") + def main(): - print("="*60) + print("=" * 60) print("Test Camera Client with Authentication") - print("="*60) + print("=" * 60) print(f"MQTT Host: {MQTT_HOST}:{MQTT_PORT}") print(f"Username: {MQTT_USERNAME}") print(f"Device Serial: {DEVICE_SERIAL}") print(f"Request Topic: {CAMERA_READ_TOPIC}") print(f"Response Topic: {CAMERA_WRITE_TOPIC}") print() - + client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="test-client-auth") client.username_pw_set(MQTT_USERNAME, MQTT_PASSWORD) client.on_connect = on_connect client.on_subscribe = on_subscribe client.on_message = on_message - + try: print("Connecting to broker...") client.connect(MQTT_HOST, MQTT_PORT, keepalive=60) client.loop_start() - + # Give it a moment to connect and subscribe time.sleep(2) - + # Send capture command msg = {"command": "capture_image"} payload = json.dumps(msg) - print(f"\nSending capture command...") + print("\nSending capture command...") print(f" Topic: {CAMERA_READ_TOPIC}") print(f" Payload: {payload}") - + result = client.publish(CAMERA_READ_TOPIC, payload, qos=1) if result.rc == 0: - print(f" ✓ Publish successful") + print(" ✓ Publish successful") else: print(f" ✗ Publish failed: {result.rc}") - + # Wait for response - print(f"\nWaiting for response (timeout: 10s)...") + print("\nWaiting for response (timeout: 10s)...") try: data = data_queue.get(True, 10) - print(f"\n✓ SUCCESS! Received response:") + print("\n✓ SUCCESS! Received response:") print(f" Image URI: {data.get('image_uri')}") print(f" Timestamp: {data.get('timestamp')}") print(f" Device: {data.get('device_serial')}") except Empty: - print(f"\n✗ TIMEOUT: No response received after 10s") - print(f" Possible issues:") - print(f" - Device not running") - print(f" - Topic ACL restrictions preventing communication") - print(f" - Credentials mismatch") - + print("\n✗ TIMEOUT: No response received after 10s") + print(" Possible issues:") + print(" - Device not running") + print(" - Topic ACL restrictions preventing communication") + print(" - Credentials mismatch") + except KeyboardInterrupt: print("\n\nShutting down...") except Exception as e: print(f"✗ Error: {e}") import traceback + traceback.print_exc() finally: client.loop_stop() client.disconnect() print("\nClient disconnected") + if __name__ == "__main__": main() diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device.py b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device.py index 9c12ab7b..67f31f50 100755 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device.py +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device.py @@ -4,9 +4,10 @@ This simulates the Raspberry Pi camera device behavior. """ import json -import paho.mqtt.client as mqtt from datetime import datetime, timezone +import paho.mqtt.client as mqtt + # Configuration MQTT_HOST = "localhost" MQTT_PORT = 1883 @@ -14,40 +15,42 @@ CAMERA_READ_TOPIC = f"rpi-zero2w/still-camera/{DEVICE_SERIAL}/request" CAMERA_WRITE_TOPIC = f"rpi-zero2w/still-camera/{DEVICE_SERIAL}/response" + def on_connect(client, userdata, flags, rc, properties=None): if rc == 0: - print(f"✓ Device connected to MQTT broker") + print("✓ Device connected to MQTT broker") print(f" Subscribing to: {CAMERA_READ_TOPIC}") client.subscribe(CAMERA_READ_TOPIC, qos=1) else: print(f"✗ Connection failed with code {rc}") + def on_message(client, userdata, msg): print(f"\n✓ Received message on topic: {msg.topic}") print(f" Payload: {msg.payload.decode()}") - + try: data = json.loads(msg.payload) command = data.get("command") - + if command == "capture_image": - print(f" Processing capture_image command...") - + print(" Processing capture_image command...") + # Simulate image capture and S3 upload timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d-%H:%M:%S") object_name = f"{timestamp}.jpeg" image_uri = f"https://test-bucket.s3.us-east-2.amazonaws.com/{object_name}" - + response = { "image_uri": image_uri, "timestamp": timestamp, - "device_serial": DEVICE_SERIAL + "device_serial": DEVICE_SERIAL, } - + payload = json.dumps(response) print(f" Publishing response to: {CAMERA_WRITE_TOPIC}") print(f" Response: {payload}") - + result = client.publish(CAMERA_WRITE_TOPIC, payload, qos=1) print(f" Publish result: {result.rc} (0=success)") else: @@ -57,20 +60,23 @@ def on_message(client, userdata, msg): except Exception as e: print(f" ✗ Error processing message: {e}") + def main(): - print("="*60) + print("=" * 60) print("Mock Camera Device Starting") - print("="*60) + print("=" * 60) print(f"MQTT Host: {MQTT_HOST}:{MQTT_PORT}") print(f"Device Serial: {DEVICE_SERIAL}") print(f"Request Topic: {CAMERA_READ_TOPIC}") print(f"Response Topic: {CAMERA_WRITE_TOPIC}") print() - - client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"device-{DEVICE_SERIAL}") + + client = mqtt.Client( + mqtt.CallbackAPIVersion.VERSION2, client_id=f"device-{DEVICE_SERIAL}" + ) client.on_connect = on_connect client.on_message = on_message - + try: client.connect(MQTT_HOST, MQTT_PORT, keepalive=60) print("Waiting for capture commands... (Ctrl+C to exit)") @@ -83,5 +89,6 @@ def main(): client.disconnect() print("Device disconnected") + if __name__ == "__main__": main() diff --git a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device_auth.py b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device_auth.py index d427b88f..7e601385 100755 --- a/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device_auth.py +++ b/src/ac_training_lab/a1_cam/_scripts/mosquitto_tests/test_mqtt_device_auth.py @@ -4,9 +4,10 @@ Tests username/password authentication and topic ACL filters. """ import json -import paho.mqtt.client as mqtt from datetime import datetime, timezone +import paho.mqtt.client as mqtt + # Configuration MQTT_HOST = "localhost" MQTT_PORT = 1883 @@ -16,9 +17,10 @@ CAMERA_READ_TOPIC = f"rpi-zero2w/still-camera/{DEVICE_SERIAL}/request" CAMERA_WRITE_TOPIC = f"rpi-zero2w/still-camera/{DEVICE_SERIAL}/response" + def on_connect(client, userdata, flags, rc, properties=None): if rc == 0: - print(f"✓ Device authenticated and connected") + print("✓ Device authenticated and connected") print(f" Username: {MQTT_USERNAME}") print(f" Subscribing to: {CAMERA_READ_TOPIC}") result = client.subscribe(CAMERA_READ_TOPIC, qos=1) @@ -29,42 +31,44 @@ def on_connect(client, userdata, flags, rc, properties=None): 2: "Connection refused - invalid client identifier", 3: "Connection refused - server unavailable", 4: "Connection refused - bad username or password", - 5: "Connection refused - not authorized" + 5: "Connection refused - not authorized", } print(f"✗ Connection failed: {error_messages.get(rc, f'Unknown error ({rc})')}") + def on_subscribe(client, userdata, mid, granted_qos, properties=None): print(f"✓ Subscription confirmed (QoS: {granted_qos})") + def on_message(client, userdata, msg): print(f"\n✓ Received message on topic: {msg.topic}") print(f" Payload: {msg.payload.decode()}") - + try: data = json.loads(msg.payload) command = data.get("command") - + if command == "capture_image": - print(f" Processing capture_image command...") - + print(" Processing capture_image command...") + # Simulate image capture and S3 upload timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d-%H:%M:%S") object_name = f"{timestamp}.jpeg" image_uri = f"https://test-bucket.s3.us-east-2.amazonaws.com/{object_name}" - + response = { "image_uri": image_uri, "timestamp": timestamp, - "device_serial": DEVICE_SERIAL + "device_serial": DEVICE_SERIAL, } - + payload = json.dumps(response) print(f" Publishing response to: {CAMERA_WRITE_TOPIC}") print(f" Response: {payload}") - + result = client.publish(CAMERA_WRITE_TOPIC, payload, qos=1) if result.rc == 0: - print(f" ✓ Publish successful") + print(" ✓ Publish successful") else: print(f" ✗ Publish failed: {result.rc}") else: @@ -74,23 +78,26 @@ def on_message(client, userdata, msg): except Exception as e: print(f" ✗ Error processing message: {e}") + def main(): - print("="*60) + print("=" * 60) print("Mock Camera Device with Authentication") - print("="*60) + print("=" * 60) print(f"MQTT Host: {MQTT_HOST}:{MQTT_PORT}") print(f"Username: {MQTT_USERNAME}") print(f"Device Serial: {DEVICE_SERIAL}") print(f"Request Topic: {CAMERA_READ_TOPIC}") print(f"Response Topic: {CAMERA_WRITE_TOPIC}") print() - - client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"device-{DEVICE_SERIAL}") + + client = mqtt.Client( + mqtt.CallbackAPIVersion.VERSION2, client_id=f"device-{DEVICE_SERIAL}" + ) client.username_pw_set(MQTT_USERNAME, MQTT_PASSWORD) client.on_connect = on_connect client.on_subscribe = on_subscribe client.on_message = on_message - + try: print("Connecting to broker...") client.connect(MQTT_HOST, MQTT_PORT, keepalive=60) @@ -101,10 +108,12 @@ def main(): except Exception as e: print(f"✗ Error: {e}") import traceback + traceback.print_exc() finally: client.disconnect() print("Device disconnected") + if __name__ == "__main__": main() diff --git a/src/ac_training_lab/a1_cam/_scripts/prefect/copilot-logs.md b/src/ac_training_lab/a1_cam/_scripts/prefect/copilot-logs.md index c204af10..c5c9d21d 100644 --- a/src/ac_training_lab/a1_cam/_scripts/prefect/copilot-logs.md +++ b/src/ac_training_lab/a1_cam/_scripts/prefect/copilot-logs.md @@ -343,7 +343,7 @@ Using "Apply Patch" ``` ``` -Patch applied — the flow now logs the captured S3 URL using `get_run_logger().info()`, so the URL will appear in the run logs (and in the Prefect UI Logs view). +Patch applied — the flow now logs the captured S3 URL using `get_run_logger().info()`, so the URL will appear in the run logs (and in the Prefect UI Logs view). Notes: - The flow return value still may not appear in the UI Outputs unless you configure result persistence; logging guarantees visibility. @@ -353,16 +353,16 @@ Made changes. User: I already added a print statement in the class itself. -I need results persistence then I guess. #context7 +I need results persistence then I guess. #context7 GitHub Copilot: I'll fetch Prefect documentation on result persistence and storage so you can persist flow outputs. -Ran `resolve-library-id` +Ran `resolve-library-id` Completed with input: { "libraryName": "prefect" } -Ran `get-library-docs` +Ran `get-library-docs` Completed with input: { "context7CompatibleLibraryID": "/prefecthq/prefect", "topic": "results persistence", @@ -615,9 +615,9 @@ PY Made changes. -User: Just look up the official docs #context7 +User: Just look up the official docs #context7 -GitHub Copilot: Ran `get-library-docs` +GitHub Copilot: Ran `get-library-docs` Completed with input: { "context7CompatibleLibraryID": "/prefecthq/prefect", "topic": "run_deployment", @@ -695,7 +695,7 @@ Pick A, B, or C and I’ll apply the changes and re-register the deployment. Made changes. -User: At this point, wouldn't I just use Prefect's S3 integration instead of boto3 to begin with? Likewise, it really seems like I shouldn't need to use S3 or log scraping to get a device on one network to receive a string from a device on another network #context7 +User: At this point, wouldn't I just use Prefect's S3 integration instead of boto3 to begin with? Likewise, it really seems like I shouldn't need to use S3 or log scraping to get a device on one network to receive a string from a device on another network #context7 GitHub Copilot: Summarized conversation historyI'll add a short todo, then fetch Prefect docs on S3 blocks and result persistence to recommend the simplest approach. @@ -703,12 +703,12 @@ Created 1 todo Getting the Prefect library doc ID first. -Ran `resolve-library-id` +Ran `resolve-library-id` Completed with input: { "libraryName": "prefect" } -Ran `get-library-docs` +Ran `get-library-docs` Completed with input: { "context7CompatibleLibraryID": "/prefecthq/prefect", "topic": "S3 blocks result persistence result_storage persist_result run_deployment", @@ -820,4 +820,4 @@ Next recommendations Which of those would you like next? -Made changes. \ No newline at end of file +Made changes. diff --git a/src/ac_training_lab/a1_cam/_scripts/prefect/device_prefect.py b/src/ac_training_lab/a1_cam/_scripts/prefect/device_prefect.py index 68e8adb5..c6ece75c 100644 --- a/src/ac_training_lab/a1_cam/_scripts/prefect/device_prefect.py +++ b/src/ac_training_lab/a1_cam/_scripts/prefect/device_prefect.py @@ -1,18 +1,17 @@ +import os from datetime import datetime, timezone from pathlib import Path -import os import boto3 from libcamera import Transform -from picamera2 import Picamera2 -from prefect import flow - from my_secrets import ( AWS_ACCESS_KEY_ID, AWS_REGION, AWS_SECRET_ACCESS_KEY, BUCKET_NAME, ) +from picamera2 import Picamera2 +from prefect import flow def _configure_picam2() -> Picamera2: @@ -66,6 +65,9 @@ def capture_image() -> str: entrypoint="src/ac_training_lab/a1_cam/device_prefect.py:capture_image", ).deploy( name="capture-image", - description="Capture a Picamera2 image and upload it to S3 using the configured bucket.", + description=( + "Capture a Picamera2 image and upload it to S3 using the configured " + "bucket." + ), work_pool_name=os.environ.get("PREFECT_WORK_POOL", "a1-cam-pool"), ) diff --git a/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator.py b/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator.py index 9f28a018..2ca8f188 100644 --- a/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator.py +++ b/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator.py @@ -12,10 +12,13 @@ import asyncio import sys -from prefect.deployments import run_deployment + from prefect.client.orchestration import get_client +from prefect.deployments import run_deployment -# WARNING: These have not been working, just simply trying to get the returned result without making an S3 bucket just for passing a string around. See src/ac_training_lab/a1_cam/_scripts/prefect/copilot-logs.md +# WARNING: These have not been working. This is trying to get the returned +# result without making an S3 bucket just for passing a string around. See +# src/ac_training_lab/a1_cam/_scripts/prefect/copilot-logs.md. async def run_and_get(deployment_name: str = "capture-image/capture-image") -> object: diff --git a/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator_context7.py b/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator_context7.py index a5af8e4c..0f6bc95d 100644 --- a/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator_context7.py +++ b/src/ac_training_lab/a1_cam/_scripts/prefect/run_capture_orchestrator_context7.py @@ -1,10 +1,10 @@ import asyncio -from uuid import UUID from typing import Optional +from uuid import UUID from prefect.client.orchestration import get_client -from prefect.flow_runs import wait_for_flow_run from prefect.deployments import run_deployment +from prefect.flow_runs import wait_for_flow_run async def capture_image_async(deployment_identifier: str, timeout: int = 300) -> str: diff --git a/src/ac_training_lab/a1_cam/device.py b/src/ac_training_lab/a1_cam/device.py index c43dad19..fd94b462 100644 --- a/src/ac_training_lab/a1_cam/device.py +++ b/src/ac_training_lab/a1_cam/device.py @@ -127,13 +127,15 @@ def on_connect(client, userdata, flags, rc, properties=None): while True: elapsed = round(time() - start_time) current_time = time() - + # Print with exponentially increasing intervals (5s, 10s, 20s, 40s, max 300s) if current_time - last_print_time >= print_interval: print(f"Running... Elapsed: {elapsed}s") last_print_time = current_time - print_interval = min(print_interval * 2, 300) # Double interval, cap at 5 minutes - + print_interval = min( + print_interval * 2, 300 + ) # Double interval, cap at 5 minutes + sleep(5) except Exception as e: diff --git a/src/ac_training_lab/a1_cam/my_secrets_example.py b/src/ac_training_lab/a1_cam/my_secrets_example.py index 03616bdd..38a39171 100644 --- a/src/ac_training_lab/a1_cam/my_secrets_example.py +++ b/src/ac_training_lab/a1_cam/my_secrets_example.py @@ -9,4 +9,6 @@ AWS_REGION = "your-region" AWS_ACCESS_KEY_ID = "your-aws-access-key-id" AWS_SECRET_ACCESS_KEY = "your-aws-secret-access-key" -IMAGE_QUALITY = 85 # JPEG quality (1-100). Lower = smaller file size. 85 gives ~2-3 MB images +IMAGE_QUALITY = ( + 85 # JPEG quality (1-100). Lower = smaller file size. 85 gives ~2-3 MB images +) diff --git a/src/ac_training_lab/a1_cam/test_camera.ipynb b/src/ac_training_lab/a1_cam/test_camera.ipynb index ac0241c4..f1f5a1e6 100644 --- a/src/ac_training_lab/a1_cam/test_camera.ipynb +++ b/src/ac_training_lab/a1_cam/test_camera.ipynb @@ -276,4 +276,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} \ No newline at end of file +} diff --git a/src/ac_training_lab/ot-2/_scripts/prefect_deploy/README.md b/src/ac_training_lab/ot-2/_scripts/prefect_deploy/README.md index 7420dd80..3fe16bc2 100644 --- a/src/ac_training_lab/ot-2/_scripts/prefect_deploy/README.md +++ b/src/ac_training_lab/ot-2/_scripts/prefect_deploy/README.md @@ -106,4 +106,4 @@ To add more devices/queues: 3. Start workers on new devices targeting specific queues 4. Update `orchestrator.py` to submit jobs to new queues -This pattern scales well for multiple laboratory devices with different capabilities and priorities. \ No newline at end of file +This pattern scales well for multiple laboratory devices with different capabilities and priorities. diff --git a/src/ac_training_lab/ot-2/_scripts/prefect_deploy/prefect.yaml b/src/ac_training_lab/ot-2/_scripts/prefect_deploy/prefect.yaml index c09cd4bf..acda3ce6 100644 --- a/src/ac_training_lab/ot-2/_scripts/prefect_deploy/prefect.yaml +++ b/src/ac_training_lab/ot-2/_scripts/prefect_deploy/prefect.yaml @@ -45,4 +45,4 @@ push: null pull: - prefect.deployments.steps.git_clone_project: repository: https://github.com/AccelerationConsortium/ac-training-lab.git - branch: main \ No newline at end of file + branch: main diff --git a/src/ac_training_lab/picam/README.md b/src/ac_training_lab/picam/README.md index 20c71e24..ea341aa3 100644 --- a/src/ac_training_lab/picam/README.md +++ b/src/ac_training_lab/picam/README.md @@ -176,7 +176,7 @@ Controls the visibility of your YouTube live stream: - `"public"` - Anyone can find and view the stream - `"unlisted"` - Anyone with the link can view, but it won't appear in search results -**Recommendation:** +**Recommendation:** - Use `"unlisted"` for lab monitoring. This provides controlled access while keeping streams discoverable by your team. - If you use `"private"`, you must provide access by adding a Google account for each person. - Each person added to the channel will be able to see all videos in the channel. diff --git a/src/ac_training_lab/picam/device.py b/src/ac_training_lab/picam/device.py index 19995805..2be98ccb 100644 --- a/src/ac_training_lab/picam/device.py +++ b/src/ac_training_lab/picam/device.py @@ -1,22 +1,37 @@ -import argparse import json -import subprocess import shutil +import subprocess import requests from my_secrets import ( CAM_NAME, CAMERA_HFLIP, + CAMERA_ROTATION, CAMERA_VFLIP, + FRAME_RATE, LAMBDA_FUNCTION_URL, PRIVACY_STATUS, + RESOLUTION, + TIMESTAMP_OVERLAY, WORKFLOW_NAME, ) +# Resolution mappings for YouTube-compatible resolutions +RESOLUTION_MAP = { + "144p": (256, 144), + "240p": (426, 240), + "360p": (640, 360), + "480p": (854, 480), + "720p": (1280, 720), + "1080p": (1920, 1080), +} + def get_camera_command(): """ - Returns the available camera command: 'rpicam-vid' (trixie) or 'libcamera-vid' (bookworm). + Returns the available camera command. + + Uses 'rpicam-vid' on trixie or 'libcamera-vid' on bookworm. """ if shutil.which("rpicam-vid"): return "rpicam-vid" @@ -28,15 +43,39 @@ def get_camera_command(): ) -def start_stream(ffmpeg_url, width=854, height=480): +def start_stream( + ffmpeg_url, width=854, height=480, rotation=0, framerate=15, timestamp_overlay=False +): """ Starts the libcamera -> ffmpeg pipeline and returns two Popen objects: p1: camera process (rpicam-vid or libcamera-vid) p2: ffmpeg process + + Args: + ffmpeg_url: RTMP URL for streaming + width: Output width in pixels (final output after rotation) + height: Output height in pixels (final output after rotation) + rotation: Rotation angle (0, 90, 180, 270 degrees clockwise) + framerate: Frame rate in fps + timestamp_overlay: Whether to show timestamp on video """ # Get the available camera command camera_cmd = get_camera_command() + # Camera always captures in landscape orientation using the full sensor. + # For portrait output (90/270 rotation), we capture landscape and rotate in ffmpeg. + # This preserves the full field of view instead of cropping. + # + # For 90/270 rotation, capture at height x width (landscape), then + # rotate to width x height (portrait). For 0/180 rotation, capture at + # width x height and output the same orientation. + if rotation in (90, 270): + # For portrait output, capture in landscape (swap dimensions for camera) + # Camera captures height x width, then ffmpeg rotates to width x height + cam_width, cam_height = height, width + else: + cam_width, cam_height = width, height + # First: camera command with core parameters libcamera_cmd = [ camera_cmd, @@ -44,14 +83,12 @@ def start_stream(ffmpeg_url, width=854, height=480): "--nopreview", "-t", "0", - "--mode", - "1536:864", # A known 16:9 sensor mode "--width", - str(width), # Scale width + str(cam_width), # Scale width "--height", - str(height), # Scale height + str(cam_height), # Scale height "--framerate", - "15", # Frame rate + str(framerate), # Frame rate "--codec", "h264", # H.264 encoding "--bitrate", @@ -67,6 +104,44 @@ def start_stream(ffmpeg_url, width=854, height=480): # Add output parameters last libcamera_cmd.extend(["-o", "-"]) # Output to stdout (pipe) + # Build video filter chain for ffmpeg + video_filters = [] + + # Add rotation filter if needed + if rotation == 90: + video_filters.append("transpose=1") # 90 degrees clockwise + elif rotation == 180: + video_filters.append("hflip,vflip") # 180 degrees + elif rotation == 270: + video_filters.append( + "transpose=2" + ) # 90 degrees counter-clockwise (270 clockwise) + + # Add timestamp overlay if enabled + # Format: YYYY-MM-DD_HH-MM-SS. ffmpeg's localtime evaluates per-frame, + # so the overlay updates once per second at typical framerates. + if timestamp_overlay: + # drawtext filter with white text, black background box, in top-left corner + # fontsize scales with video height for consistent appearance + if rotation in (90, 270): + actual_height = width + else: + actual_height = height + fontsize = max(16, actual_height // 20) + # Note: In ffmpeg drawtext filter, special characters need escaping: + # - The format separator after localtime uses \: + # - Colons in the time display (H:M:S) also need escaping + # - Using dashes instead of colons in time to avoid complex escaping issues + timestamp_filter = ( + f"drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf" + f":fontsize={fontsize}" + f":fontcolor=white" + f":box=1:boxcolor=black@0.5:boxborderw=5" + f":x=10:y=10" + f":text='%{{localtime\\:%Y-%m-%d_%H-%M-%S}}'" + ) + video_filters.append(timestamp_filter) + # Second: ffmpeg command ffmpeg_cmd = [ "ffmpeg", @@ -83,24 +158,42 @@ def start_stream(ffmpeg_url, width=854, height=480): # Read H.264 video from pipe "-i", "pipe:0", - # Copy the H.264 video directly - "-c:v", - "copy", - # Encode audio as AAC - "-c:a", - "aac", - "-b:a", - "128k", - "-preset", - "fast", - "-strict", - "experimental", - # Output format is FLV, then final RTMP URL - "-f", - "flv", - ffmpeg_url, ] + # Add video filter and encoding settings + # Note: When filters are applied, libx264 encoding is required which increases + # CPU usage compared to the original H.264 passthrough. This is unavoidable + # since ffmpeg cannot apply filters without re-encoding the video stream. + if video_filters: + filter_chain = ",".join(video_filters) + ffmpeg_cmd.extend( + ["-vf", filter_chain, "-c:v", "libx264", "-preset", "ultrafast"] + ) + else: + ffmpeg_cmd.extend(["-c:v", "copy"]) + + ffmpeg_cmd.extend( + [ + # Encode audio as AAC + "-c:a", + "aac", + "-b:a", + "128k", + "-strict", + "experimental", + # Fix non-monotonous DTS warnings by enabling audio synchronization + # and constant frame rate video output + "-async", + "1", + "-vsync", + "cfr", + # Output format is FLV, then final RTMP URL + "-f", + "flv", + ffmpeg_url, + ] + ) + # Start camera process, capturing its output in a pipe p1 = subprocess.Popen( libcamera_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT @@ -150,27 +243,53 @@ def call_lambda(action, CAM_NAME, WORKFLOW_NAME, privacy_status="private"): raise RuntimeError(f"Failed to decode Lambda response: {e}") +def end_previous_broadcast(): + try: + call_lambda("end", CAM_NAME, WORKFLOW_NAME) + except RuntimeError as e: + print(f"Could not end previous broadcast; continuing to create a new one: {e}") + + if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Stream camera feed via Lambda") - parser.add_argument( - "--resolution", - type=str, - default="854x480", - help="Camera resolution as WIDTHxHEIGHT (default: 854x480)" + # Validate and get resolution + if RESOLUTION not in RESOLUTION_MAP: + raise ValueError( + f"Invalid RESOLUTION '{RESOLUTION}'. " + f"Allowed options: {list(RESOLUTION_MAP.keys())}" + ) + width, height = RESOLUTION_MAP[RESOLUTION] + + # Validate rotation + if CAMERA_ROTATION not in (0, 90, 180, 270): + raise ValueError( + f"Invalid CAMERA_ROTATION '{CAMERA_ROTATION}'. " + f"Allowed options: 0, 90, 180, 270" + ) + + # Validate frame rate + if not isinstance(FRAME_RATE, int) or FRAME_RATE <= 0: + raise ValueError( + f"Invalid FRAME_RATE '{FRAME_RATE}'. " + f"Must be a positive integer (e.g., 15, 24, 30)" + ) + + # For 90/270 rotation, output is portrait (swapped dimensions) + if CAMERA_ROTATION in (90, 270): + output_width, output_height = height, width + orientation = "portrait" + else: + output_width, output_height = width, height + orientation = "landscape" + + print( + f"Using resolution: {RESOLUTION} ({output_width}x{output_height} {orientation})" ) - args = parser.parse_args() - - # Parse resolution - try: - width, height = map(int, args.resolution.split('x')) - except ValueError: - print(f"Invalid resolution format: {args.resolution}. Use WIDTHxHEIGHT format.") - exit(1) - - print(f"Using resolution: {width}x{height}") - + print(f"Using rotation: {CAMERA_ROTATION} degrees") + print(f"Using frame rate: {FRAME_RATE} fps") + print(f"Timestamp overlay: {'enabled' if TIMESTAMP_OVERLAY else 'disabled'}") + # End previous broadcast and start a new one via Lambda - call_lambda("end", CAM_NAME, WORKFLOW_NAME) + end_previous_broadcast() raw_body = call_lambda( "create", CAM_NAME, WORKFLOW_NAME, privacy_status=PRIVACY_STATUS ) @@ -186,7 +305,9 @@ def call_lambda(action, CAM_NAME, WORKFLOW_NAME, privacy_status="private"): while True: print("Starting stream..") - p1, p2 = start_stream(ffmpeg_url, width, height) + p1, p2 = start_stream( + ffmpeg_url, width, height, CAMERA_ROTATION, FRAME_RATE, TIMESTAMP_OVERLAY + ) print("Stream started") interrupted = False try: diff --git a/src/ac_training_lab/picam/my_secrets_example.py b/src/ac_training_lab/picam/my_secrets_example.py index c5b8d645..0bbe541c 100644 --- a/src/ac_training_lab/picam/my_secrets_example.py +++ b/src/ac_training_lab/picam/my_secrets_example.py @@ -30,3 +30,38 @@ CAMERA_VFLIP = True # Set to True to flip the camera image horizontally (mirror image) CAMERA_HFLIP = True + +# Camera rotation setting (for portrait mode streaming) +# Allowed options: 0, 90, 180, 270 (degrees, clockwise) +# Default: 0 (no rotation / landscape mode) +# Use 90 or 270 for portrait mode streaming +CAMERA_ROTATION = 0 + +# Stream resolution setting +# Allowed options for YouTube: "144p", "240p", "360p", "480p", "720p", "1080p" +# Resolution mappings: +# "144p" = 256x144 +# "240p" = 426x240 +# "360p" = 640x360 +# "480p" = 854x480 +# "720p" = 1280x720 +# "1080p" = 1920x1080 +# Default: "480p" +# Note: Pi Zero 2W can comfortably handle 480p at 15fps. 720p at 15fps is pushing it. +# For 1080p or higher frame rates, use a Pi 4B or Pi 5. +RESOLUTION = "480p" + +# Stream frame rate setting (frames per second) +# Common values: 15, 24, 30 +# Default: 15 +# Note: Pi Zero 2W can comfortably handle 15fps at 480p. +# For higher frame rates or resolutions, use a Pi 4B or Pi 5. +FRAME_RATE = 15 + +# Timestamp overlay setting +# Set to True to display current date/time on the video stream +# Format: YYYY-MM-DD_HH-MM-SS (e.g., 2024-12-01_21-38-37) +# The timestamp appears in the top-left corner (white text with black background) +# Note: Enabling timestamp requires video re-encoding which increases CPU usage. +# Default: False +TIMESTAMP_OVERLAY = False