Here is a notebook for automating the download, filter, and prep for normalization process for Monthly Patterns data in Google Colab.
It’s designed to make things as easy as possible for you. All you have to do is change a couple variables to match your setup, define how you want to filter the data, and run the script. It is storage-efficient and fast. For each month of data you process, it takes just over 3 minutes to download to Google Drive, filter, and organize. Perhaps best of all, the resulting data in your Google Drive is ready for use with @Ryan_Fox_Squire_SafeGraph’s notebook for best practices in normalization.
Feedback and ideas for improvement are welcome!
Hi Ryan! Thank you very much about this example. I’m a student from Harvard and I’m new to s3 and safegraph. Could you explain more about this file “core/2020/06/Core-USA-June2020-Release-CORE_POI-2020_05-2020-06-06.zip”? Do we need to change the address if we want the latest data? Many thanks!
Hi @Xiaohan_Yang, you are welcome! I just updated the notebook (like two seconds ago!) to pull the October Core POI file instead of June. November is the most recent, but the delivery method changed in November, so it will take a bigger update before the notebook is able to pull the newest month’s POIs. I do not expect to make the bigger update for some time, but let me know if you need it.
November and October should not have many differences. Also, October’s POIs have a column for placekey, which will come in handy if you need to join with other data. Is this okay?