Skip to content

Refactor pisces#1

Open
p-galligan wants to merge 26 commits intodevelopmentfrom
refactor-pisces
Open

Refactor pisces#1
p-galligan wants to merge 26 commits intodevelopmentfrom
refactor-pisces

Conversation

@p-galligan
Copy link
Member

@p-galligan p-galligan commented Feb 17, 2026

Adds code and tests.

These aren't passing tests right now, and I think it has to do with some of the mappings. I also don't currently have tests for the lambda_handler.

You'll notice I have some environment setting. I didn't really get to mess with pulling from SSM, so that still needs work.

I think there are also some fixtures or schemas that could be removed and that aren't being used right now.

To Do:

  • Update code to pull environment variables from SSM for runtime
  • Update License and Readme documentation
  • Remove unnecessary code from fetchers/helpers/mergers/resources
  • Write tests for lambda_handler function
  • Increase code coverage for mappings and transformers
  • Create CloudFormation template

Adds initial application structure
Adds more files
Adds test requirements. Tweaks mappings and transformers.
Adds coveragerc file
Updates files to correct linting errors.
Fixes linting
Adds tests and fixtures
@p-galligan p-galligan requested review from helrond and ioduok February 17, 2026 21:57
Copy link
Member

@helrond helrond left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some specific notes but a couple of global things:

  • somehow all the pycache stuff found its way into version control, which makes things a lot messier than they should be. Can you get this stuff out of VCS?
  • I think my next move would be to move the helpers that are still needed out of the src/fetcher directory and then just delete all that stuff.
  • It looks to me like config handling is going to be somewhat complex in this app, so it might be worth thinking about what you want as an environment variable (i.e. what is specific to that particular lambda instance) and what will be the same across all instances (like credentials or common configs) and should consequently be pulled from SSM.

Removes old pycache files.
Removes iso639, implements pycountry, updates requirements, and updates dependency actions.
fixes current tests
Accidentally passed online_pending along in transformation.
Removes unnecessary logic
Fixes linting and ignores boolean at start of line warning in tox (this is a modern accepted practice).
@p-galligan
Copy link
Member Author

I also didn't include this from the old transformer because I wasn't sure we still needed it, since we're not saving anything, right? Or should I put this back into the transformer?

https://github.com/RockefellerArchiveCenter/pisces/blob/71ed3b5e77c86614857bda186e760ed51cb9abd5/transformer/transformers.py#L109-L123

Fixes transformer and test to actually use schemas (whoops).
Updates mappings and transformers to adhere strictly to old Pisces mappings as close as possible. Remove any normalization from transformers.
Implements functionality to get environment from ssm
Adds tests to cover more statements in transformers.
Updates test mappings. Removes old comment
Adds license and readme.
Tweaks transformer comments
Updates mapping tests to not call the environment import each function.
Copy link
Member

@HaSistrunk HaSistrunk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few minor things, I haven't delved deeply into the code. Are tests not running on your commits?

p-galligan and others added 2 commits February 27, 2026 10:23
Cleans up deploy and gitignore and updates test_requirements
Fixes typo

Co-authored-by: Hannah Sistrunk <hsistrunk@rockarch.org>
@p-galligan p-galligan closed this Feb 27, 2026
@p-galligan p-galligan reopened this Feb 27, 2026
@p-galligan
Copy link
Member Author

Just a few minor things, I haven't delved deeply into the code. Are tests not running on your commits?

Thanks! I had the wrong name for the workflows folder. And this uncovered an issue that I didn't know what happening locally. (My CLI had variables but the workflow environment doesn't, so tests fail. Gotta fix that)

Fixes sns client to make it not require AWS during CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants