ClaytonWWilson/Listify
1
0
Fork 0
mirror of https://github.com/ClaytonWWilson/Listify.git synced 2026-03-10 18:55:03 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Chaining, retries, and general scraping durability

Browse Source 
Make the scraping infrastructure more durable since the scraping service often fails to deliver
This commit is contained in:
NMerz
2020-11-01 11:30:03 -05:00
parent 79598bf9e9
commit ee04a06e0e
21 changed files with 74 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
Lambdas/Scraping/buildRunOrchestratorZip.sh Normal file
View File

@@ -0,0 +1,5 @@
#!/bin/bash
#Currently to be run only from the Scraping directory
rm artifacts/runOrchestrator.zip
zip -r9 artifacts/runOrchestrator.zip prefix_list_part*.txt
zip -r9 artifacts/runOrchestrator.zip runOrchestrator.py