Do I need a key for the AWS CLI?
ok great - so you will likely find the SafeGraph_py library helpful in your munging / reading in process
Perfect
Is the Safegraph visit data available for all of 2020?
yes! 2018, 2019, 2020, jan feb 2021
There is data that was recorded in the time and then backfilled data (every 6 months or so there is a backfill) the backfill attempts to eliminate bugs and weird artifacts from the data, as well as update it with any new columns added. I recommend using the backfill whenever possible.
Check out the FAQ for more information of stuff like that
I don’t meant o send you down rabbit holes, but I assume there will be questions haha (always feel free to ask here)
Maybe it’s because I’m just starting to look at the data, but how do you select multiple variables and download a sample in the webportal? I can only select one variable at a time. Does this makes sense
You cannot pre filter the data before you pull it strictly using SafeGraph products. However, there are some third parties that try to make this possible like <#C0139RQSTRP|rill-support> or <#C012T1Z8FQA|unfolded-studio-support>, but besides that your only option is to pull the data and then filter it to a subset you want.
there is also a notebook developed by @Ryan_Kruse_MN_State that can help filter the data as you download it using boto3
Are there unique keys that would allow merging between different data tables? I am assuming that I can download daily mobility data and then merge that with a table containing any missing variables? Also, what do you recommend doing this merger on? Python, Power BI?
correct, you can merge on Safegraph_place_id
that will allow a pattern to core place merge. If you decide to merge to data outside of safegraph, you can placekey that data and merge on Placekeys
I use python - seems to be the fastest outside of something like SCALA or Spark or a SQL server (you can use mysql as well).
So you can merge the data without using a third party app?
internal safegraph data, yes.
if you want to merge safegraph to say government permit data, you would add placekeys to the govt data and merge to safegraph using Placekey (it already has placekeys on it)
I believe all the merging I want to do is around safegraph data. I need an observation to have the CBG, day-mobility entry for each NAICS code. Is this an operation done within the catalog?
sure. you will merge movement to core then groupby NAICS with the sum of visits – getting the daily is gonna take some computation power because it is embedded in a JSON within the weekly patterns data
Weekly -> core on safegraph place id
filter to NAICS
get sum of weekly visits
OR – explode on visit by day column and sum accordingly
Ok, but I don’t see any “merge” options when selecting the Web option within the catalog. I can only select, say Core, and nothing else. What am I missing here?
You download the core, then you download the weekly then you merge it on your computer - safegraph does not offer any: munging, merging, filtering, cutting, appending or anyhting
just a set of data that you can merge after the fact
Sorry, I got pulled away but thanks for the answer, that clears things up!
Could I reach out if I have any more questions?
No worries @Jonnathan_Cuji_University_of_Notre_Dame and absolutely, though I recommend you post a new question in a thread (higher chance I or someone else will see it).
Any data questions put in #safegraphdata and any data download questions put in #aws-troubleshooting
Feel free to ping me anytime! Good luck!