From 415ebe6a3d85a69ed9ec2999f3bf94fa4893b24c Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Fri, 20 Nov 2020 13:38:47 -0500 Subject: [PATCH 01/16] Tooling documentation --- Tooling/README.md | 68 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 Tooling/README.md diff --git a/Tooling/README.md b/Tooling/README.md new file mode 100644 index 0000000..7192c49 --- /dev/null +++ b/Tooling/README.md @@ -0,0 +1,68 @@ +# Tooling + +This folder contains supporting scripts to automate the serverless backend resource creation/updating. + + +## Scope +The scripts here will create and update AWS Lambdas and their paired API Gateways as well as attaching appropriate IAM roles to the Lambdas and Cognito integration to the API Gateway. + +Covered: + - AWS Lambdas + - AWS API Gateway + - AWS IAM <-> AWS Lambda interaction + - AWS Cognito <-> API Gateway interaction + +Not covered: + - Verification of Lambda correctness: These scripts will in no way verify that the built jar is correct. + + + +## Files + +#### EndpointSetup.sh + - Purpose: Creates a new AWS Lambda, and, if needed, a new API Gateway endpoint and HTTP method on the endpoint. The names are autogenerated based on the method and the input name. + - Inputs: + - Resource name: Name of the resource (e.x. List). + - HTTP method: HTTP method to use to access this particular Lambda. Will be appended to resource name to get expected Java class (e.x. resourceGET) + - Resource jar: The script will search the project directory for a jar with the given resource name (resource.jar). The jar should contain all method classes for the resource as well as any dependencies. + +#### LambdaUpdate.sh + - Purpose: Refreshes the jar attached to a specific Lambda. + - Inputs: + - Resource name: Name of the resource (e.x. List). + - HTTP method: HTTP method to use to access this particular Lambda. Will be appended to resource name to get expected Java class (e.x. resourceGET) + +#### VarSetup.sh + - Purpose: Sets up some constants related to the AWS account and basic resources (e.x. broad API gateway id, Cognito pool). Collects constants for easy modification + + + + +## Running + +``` +$ bash Tooling/EndpointSetup.sh +Creating a Gateway/Lambda pair. +Please enter base function name: List +Please enter method(GET, POST, etc.): PUT + +``` + +``` +$ bash Tooling/LambdaUpdate.sh +Updating a Lambda. +Please enter base function name: List +Please enter method(GET, POST, etc.): PUT +Update successful. +``` + +VarSetup.sh should not need to be called directly since the other scripts `source` it. + +## Debugging + + VarSetup.sh has a `DEBUGFILE` constant in it. By default it is `/dev/null`, but you can change it to something else to get debugging output. + +## Notes +- Many configurations are stored as constants in the scripts and can be modified. Most of these constants should be in VarSetup.sh, but some may be elsewhere. +- Sometimes the allocated memory and/or runtime is insufficient and must be adjusted upwards. +- If in Windows Subsystem for Linux, you may have to remove '\r' characters that Windows added to the .sh files. From 13f9c4ba1de17f3a37f5ce6ac34050292b6d8d35 Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Fri, 20 Nov 2020 14:00:24 -0500 Subject: [PATCH 02/16] General Lambda root readme --- Lambdas/README.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 Lambdas/README.md diff --git a/Lambdas/README.md b/Lambdas/README.md new file mode 100644 index 0000000..507841a --- /dev/null +++ b/Lambdas/README.md @@ -0,0 +1,35 @@ + +## Components +### APIs +Contains logic to retrieve data from merchant APIs. + +### Lists +Contains client-facing Lambdas which are typically a rather thin wrapper over the database (MariaDB on AWS's RDS service) and other AWS services. +#### General Client-facing Serverside Flow +``` +Client -> API Gateway endpoint -> Lambda -> (same) API Gateway endpoint -> (same) Client + | . | . + \ / / \ \ / / \ + ' | ' | + AWS Cognito MariaDB instance/Other AWS services(SES, Cognito) +``` + +### Scraping +Contains logic to orchestrate runs using a scraping utility to collect merchant data and store it in the database + + + + +## Common Troubleshooting +#### Logs +- Logs may be written to by printing to the stdout. (Fancier logging is also possible, but currently not implemented) +- Logs are avalaible on AWS's Cloudwatch under Log Groups -> {Lambda name} +- Logs typically persist after deletion of the associated Lambda + +#### Incomplete runs +Often if a run fails to complete, it is because the standard allocated time and memory/cpu is inadaquate. +- Symptoms: + - The running time (available in the Cloudwatch logs) is almost exactly the maximum time. +- Resolutions: + - Time may be increased. + - CPU may be increased. CPU is proportional to memory, so CPU is increased by increasing memory allocation. From 25faa54a5940185f01c48bdbc5b7e3f824e71846 Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Fri, 20 Nov 2020 14:35:25 -0500 Subject: [PATCH 03/16] Client-Facing Lambdas overview --- Lambdas/Lists/README.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 Lambdas/Lists/README.md diff --git a/Lambdas/Lists/README.md b/Lambdas/Lists/README.md new file mode 100644 index 0000000..aabd681 --- /dev/null +++ b/Lambdas/Lists/README.md @@ -0,0 +1,15 @@ +# Client-Facing Lambdas +Each subfolder (except `target`) is a module. + +NOTE: `src` contains the Core module + +## Building +- The pom.xml has dependency information for all the Modules combined and all these dependencies are currently built into the Core module. +- Other build configurations are done in IntelliJ with each module having a build definition. + - Each module relies the (extracted) output of Core.jar being included in its build. + - Other modules may be included as applicable. + - Some modules specifically include their own dependencies, but this is not strictly necessary since everything is (inefficiently) bundled into Core. + +## Inclusion in Git +- Compiled output should never be included in git. +- IntelliJ's build files may not be properly included (or may just be hard to import.) Copying the folder directly between machines works. From f54978dddbe1526df134f0c4d3da0ec4633c6da7 Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Fri, 20 Nov 2020 15:49:20 -0500 Subject: [PATCH 04/16] Core Module readme --- Lambdas/Lists/src/README.md | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 Lambdas/Lists/src/README.md diff --git a/Lambdas/Lists/src/README.md b/Lambdas/Lists/src/README.md new file mode 100644 index 0000000..ac76c40 --- /dev/null +++ b/Lambdas/Lists/src/README.md @@ -0,0 +1,36 @@ +# Core Module +Wraps inner business functionality and abstracts certain environment interactions such as opening/closing the database connection and processing the API Gateway input json. + +#### Expected input json: +The following is created by APIGateway along the lines of [this](https://github.com/ClaytonWWilson/Listify/blob/master/Tooling/body_and_auth_mapping.json) definition +``` +{ + "body": {jsonizedPOSTEDObject}, + "params": { + "querystring": { + "queryParamifApplicable": "paramValue" + } + }, + "context": { + "sub": "cognitoID" + } +} +``` + +`querystring` will only have subcomponents if there are query parameters + + +`{jsonizedPOSTEDObject}` will be something like: +``` +{ + "var1": 1, + "var2": "string" +} +``` + +#### Module Contract: +For the module contract see [CallHandler](https://github.com/ClaytonWWilson/Listify/blob/master/Lambdas/Lists/src/main/java/CallHandler.java) + +#### Returns +Business logic return or error is appropriate + From d56fed35a098e4b27383e92075cf0edb36f9f6de Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Thu, 3 Dec 2020 20:43:43 -0500 Subject: [PATCH 05/16] Create Chain module readme --- Lambdas/Lists/Chain/src/README.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 Lambdas/Lists/Chain/src/README.md diff --git a/Lambdas/Lists/Chain/src/README.md b/Lambdas/Lists/Chain/src/README.md new file mode 100644 index 0000000..6feafab --- /dev/null +++ b/Lambdas/Lists/Chain/src/README.md @@ -0,0 +1,15 @@ +# Chain Module +Deals with information concerning store chains supported by the product populating and product serving Lambdas. + +### ChainGET +#### Expected request body: +N/A + +#### Expected query parameters: + - id + - Used for + - Valid values: + +#### Inputs and outputs: + - id = -1: Returns an array if chainIDs (Integers) + - id = 1 Date: Thu, 3 Dec 2020 20:52:33 -0500 Subject: [PATCH 06/16] Create README.md --- Lambdas/Lists/Item/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 Lambdas/Lists/Item/README.md diff --git a/Lambdas/Lists/Item/README.md b/Lambdas/Lists/Item/README.md new file mode 100644 index 0000000..79f17f3 --- /dev/null +++ b/Lambdas/Lists/Item/README.md @@ -0,0 +1,18 @@ +# Item Module +Deals with retrieving product information populated by the scripts at https://github.com/ClaytonWWilson/Listify/blob/master/Lambdas/APIs and https://github.com/ClaytonWWilson/Listify/tree/master/Lambdas/Scraping. + +### ItemGET +#### Expected request body: +N/A + +#### Expected query parameters: + - id + - Used for specifying which product to retrieve. This is not the upc, but is our own internal id. + - Valid values: 1 Date: Thu, 3 Dec 2020 20:52:57 -0500 Subject: [PATCH 07/16] Make link point at master --- Lambdas/Lists/Chain/src/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lambdas/Lists/Chain/src/README.md b/Lambdas/Lists/Chain/src/README.md index 6feafab..a1904b8 100644 --- a/Lambdas/Lists/Chain/src/README.md +++ b/Lambdas/Lists/Chain/src/README.md @@ -12,4 +12,4 @@ N/A #### Inputs and outputs: - id = -1: Returns an array if chainIDs (Integers) - - id = 1 Date: Thu, 3 Dec 2020 20:54:20 -0500 Subject: [PATCH 08/16] Finish specifying parameter values for Chain module --- Lambdas/Lists/Chain/src/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Lambdas/Lists/Chain/src/README.md b/Lambdas/Lists/Chain/src/README.md index a1904b8..84024a6 100644 --- a/Lambdas/Lists/Chain/src/README.md +++ b/Lambdas/Lists/Chain/src/README.md @@ -7,9 +7,10 @@ N/A #### Expected query parameters: - id - - Used for - - Valid values: + - Used for specifying which chain to retrieve + - Valid values: -1,1 Date: Thu, 3 Dec 2020 21:01:42 -0500 Subject: [PATCH 09/16] Add ItemSearch readme --- Lambdas/Lists/ItemSearch/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 Lambdas/Lists/ItemSearch/README.md diff --git a/Lambdas/Lists/ItemSearch/README.md b/Lambdas/Lists/ItemSearch/README.md new file mode 100644 index 0000000..b87c856 --- /dev/null +++ b/Lambdas/Lists/ItemSearch/README.md @@ -0,0 +1,17 @@ +# ItemSearch Module +Deals with retrieving aggregate product information meeting certain naming criterion from among that populated by the scripts at https://github.com/ClaytonWWilson/Listify/blob/master/Lambdas/APIs and https://github.com/ClaytonWWilson/Listify/tree/master/Lambdas/Scraping. + +### ItemGET +#### Expected request body: +N/A + +#### Expected query parameters: + - id + - Used for specifying a substing of the product name that returned items should mathc + - Valid values: {any_string} + +#### Inputs and outputs: + - id = {any string}: Returns an [ItemSearch object](https://github.com/ClaytonWWilson/Listify/blob/master/Lambdas/Lists/ItemSearch/src/ItemSearch.java), which is basically an array of productIDs, for products whose name fits the regex criterion `name = ".*" + id + ".*"` + - Results are limited to the first 100 entries from the database (by earliest add time) + + From a81eed79465b2ba95fb5eec70daf1e2b039f3ca6 Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Thu, 3 Dec 2020 21:06:28 -0500 Subject: [PATCH 10/16] Spelling fix --- Lambdas/Lists/ItemSearch/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Lambdas/Lists/ItemSearch/README.md b/Lambdas/Lists/ItemSearch/README.md index b87c856..292b1ca 100644 --- a/Lambdas/Lists/ItemSearch/README.md +++ b/Lambdas/Lists/ItemSearch/README.md @@ -7,7 +7,7 @@ N/A #### Expected query parameters: - id - - Used for specifying a substing of the product name that returned items should mathc + - Used for specifying a substing of the product name that returned items should match - Valid values: {any_string} #### Inputs and outputs: From 754935171ecb1353c3ea15dd6b6b56817df96c88 Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Thu, 3 Dec 2020 21:25:52 -0500 Subject: [PATCH 11/16] Create List module readme --- Lambdas/Lists/List/README.md | 85 ++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 Lambdas/Lists/List/README.md diff --git a/Lambdas/Lists/List/README.md b/Lambdas/Lists/List/README.md new file mode 100644 index 0000000..6eaf910 --- /dev/null +++ b/Lambdas/Lists/List/README.md @@ -0,0 +1,85 @@ +# List Module +Deals with managing the user-facing list creation, deletion, retrieval, and renaming. The supporting classes here also serve as resources for other modules. + +### Notable +[ListPermission.java](https://github.com/ClaytonWWilson/Listify/blob/master/Lambdas/Lists/List/src/ListPermissions.java) contains the lambda-side translation from permissionLevel to actual permissions. Keys updated on the client side must also be updated here. + +### ListGET +Retrieves details for an existing list +#### Expected request body: +N/A + +#### Expected query parameters: + - id + - Used for specifying which list to retrieve + - Valid values: -1,1 Date: Thu, 3 Dec 2020 21:35:49 -0500 Subject: [PATCH 12/16] Create ListEntry module readme --- Lambdas/Lists/ListEntry/README.md | 51 +++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 Lambdas/Lists/ListEntry/README.md diff --git a/Lambdas/Lists/ListEntry/README.md b/Lambdas/Lists/ListEntry/README.md new file mode 100644 index 0000000..bdc86b3 --- /dev/null +++ b/Lambdas/Lists/ListEntry/README.md @@ -0,0 +1,51 @@ +# ListEntry Module +Deals with managing entries in a list. Note that there is no GET here because that is handled in ListGET see [List readme](https://github.com/ClaytonWWilson/Listify/tree/master/Lambdas/Lists/List) + +### ListEntryPOST +Add a specified quanitity of an item to a list. +#### Minimum expected request body: +``` +{ + "productID": 2, + "listID": 4, + "quantity": -3, + "purchased": false +} +``` +Other fields may be populated and are simply ignored. + +#### Expected query parameters: +N/A + +#### Inputs: + - productID: The productID to add to the list + - listID: The listID of the list to be modified + - quantity: The number of an item to add (may be negative) to the list + - purchased: Currently unused field with future potential for allowing user to check off items they have purchased + +#### Returns: +null + + +### ListEntryDELETE +Removes the specified product from the list + +#### Minimum expected request body: +``` +{ + "listID": 6, + "productID": 5 +} +``` +Other fields may be populated and are simply ignored. + +#### Expected query parameters: +N/A + +#### Inputs: + - listID: The listID of the list to be modified + - productID: The productID to remove from the list + +#### Returns: +null + From dd2a8413a960f9f8670d2449b6ce97738bbac8d0 Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Thu, 3 Dec 2020 21:49:13 -0500 Subject: [PATCH 13/16] ListShare module readme --- Lambdas/Lists/ListShare/README.md | 49 +++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 Lambdas/Lists/ListShare/README.md diff --git a/Lambdas/Lists/ListShare/README.md b/Lambdas/Lists/ListShare/README.md new file mode 100644 index 0000000..0c509d7 --- /dev/null +++ b/Lambdas/Lists/ListShare/README.md @@ -0,0 +1,49 @@ +# ListShare Module +Deals with permissions for accessing Lists and performing functions on them. This enabled social-media-like functionality. + +### Concerns +More information is public here than may be desirable. In the future, it may be worth reducing the amount of information served by this Lambda about users other than the requestor. + + +### ListShareGET +Retrieves a information on who may access a list and how they interact with it. +#### Expected request body: +N/A + +#### Expected query parameters: + - id + - Used for specifying which list to retrieve share information for + - Valid values: -1 Date: Thu, 3 Dec 2020 22:03:04 -0500 Subject: [PATCH 14/16] Create User module readme --- Lambdas/Lists/User/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 Lambdas/Lists/User/README.md diff --git a/Lambdas/Lists/User/README.md b/Lambdas/Lists/User/README.md new file mode 100644 index 0000000..b697098 --- /dev/null +++ b/Lambdas/Lists/User/README.md @@ -0,0 +1,26 @@ +# User Module +Wraps certain Cognito functionality to provide greater ease of use for the developer than always calling Cognitor directly and using service hooks. + +### UserGET +Retrieves information on a user's cognitoID or email. This Lambda exists for ease of use in retrieving/converting between returns. + +#### Potential Inputs (choose 1) + - (query paramter) id: A user email or a Cognito id. Note: relies on the assumption that Cognito ids do not have "@" and emails do + - (body element) emailToCheck: A user email + - No input (or null/empty string inputs) + + +#### Returns: + - A User object (if asking for an email, the Cognito id may be incorrect) + - No input results in only the Cognito id being populated with the requestor's id. + - Should something about the contract be broken, or the user not exist, a null, or null-populated, value will be returned + + +### UserDELETE +Deletes a user and their data. This Lambda exists to ease database cleanup and sending a farewell email. + +#### Inputs: +None, this only uses the Congito id collected by AWS Gateway + +#### Returns: +null From a57e4888ce40dfa530266c1280ed7d1e6ef18dba Mon Sep 17 00:00:00 2001 From: Nathan Merz Date: Thu, 3 Dec 2020 22:03:59 -0500 Subject: [PATCH 15/16] Delete .DS_Store --- .DS_Store | Bin 6148 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 .DS_Store diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index 587aaadfd88b40e1bd9de9d56ba4276786654ca7..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6148 zcmeH~-EPw`6vvO(!ki*RFOW8Il^Y~33ToNZPC`go=`JRLgvbOJfVPB`E>adxO-i+e zqTKKhe7pdUf_MNPga?4XKR`>yb_odiRQBKYImeZMO6-`3*l-s0h}uNNQKU9osD5M8 zFUu)gG9wO@$vMP}FcBeoayo04fFP&a*?aH+&8clQ_$}-LGt^(Y$bRne!HJU2Z-1_r=`LgM5|^gXvS*dl=;V zp=s|3k^E{0;p2Eb-)&#LC(=BK)A3X#$I%!e&z{6-B<4dgOQTH9<7)@!4tI9jt3}b@ z=y%+;-t9%lEe3;b$6a6RFBT48x!$|Gy?1bQT)bYqd8d|53|pJBD@K3Hdo*fLUis5B z5$OaTnI*k3Wt5|SNk>&=Yy}Z@z$cuO( zYS0{dL|e2;BYFV6g(ohmfqRQNs^E4=;Fkb%N|Rdjw^}s1))WErVDn3shb3SMSOPTy za({49q#tt{i}RMoX<2fiay5)v3%}F_^hKFr5y+XW~n( zI-QvMXUt>%nVB03GglAhs&HbSR$E&FmOzz2LwDQq{NFnJ{$EY9Jxjn6_^$}C=Exrn zu_SZ0o?4tdYh9E*6lpT9)T&ZYnd4YB Date: Thu, 3 Dec 2020 22:04:30 -0500 Subject: [PATCH 16/16] Delete .DS_Store --- Lambdas/.DS_Store | Bin 6148 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 Lambdas/.DS_Store diff --git a/Lambdas/.DS_Store b/Lambdas/.DS_Store deleted file mode 100644 index d4384bd08e802ff378a0dd88e2df30022a70988b..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6148 zcmeHK!HN?>5UtK8+KGrc2rHiEB6x_9uuF0fVT|h`;9+4z4=TGeo3J}DPKV6Iup1-u zANm7+iyz?U=&SB_*%6g?S@I=SX;T{@fUo7afOZ#(JaqQehM#VXo~2B zW)#yDgynR)PORER%vEC16%hu60byX94EW1U-`eJ)vX(F)4Ez%e@cj@#8AFeiLwj|g zu_XYo4!0H9_SYX6umc!+tQ^7vQ7#qeQk5?;l&>6=OUHQV<12?Qos?Y}^Z1pOuTYdP zJXB!ANreuj6$XTXbq01^x6SYW^Xu#X^(5&D1H!<6#ek|GL`~e%l&|~Eg9*F!1SQ?}e2DZw;PrD6tp8x;=