A question for Safegraph community both users and developers: I am trying to reformat the Weekly Patterns data using visits_by_day and visitor_home_cbgs to convert the data into daily visits from a cbg for a subset of businesses. I am using -split-, -gcollapse- and -reshape- in Stata to manipulate the data on the weekly basis. However, for each week, this manipulation takes over 10 hours to run. Does Safegraph have any shortcuts for working with the JSON fields, or any advice on how to work with this data so that it is manageable? I would appreciate any advice, methods, or suggestions for this type of data manipulation.
This topic was automatically generated from Slack. You can find the original thread here.
Hi again . Hope all is well since we last chatted during your research seminar
Just to keep you updated, I’m bringing in someone from our Product team. They would likely be able to better guide you on this than I can. Thanks, Shooshan!
, if the tools from the package are giving a long runtime, check out this method that I used in my last data project. I think this will help you do what you’re looking for. The code uses spark to expand the columns and does the job fairly quickly.
a lot of the code in there pertains to the project that was being worked on. For you needs look at the setup tab and the horizontal expansion of json columns tab. Hopefully this helps
Thank you . I am trying to work through this code, to see how to take advantage of this within the my data cleaning process. If you do not mind, I will reach back out to you with questions!
Hey ! Just confirming that we answered your question. I’m going to go ahead and close this thread out. If you have any more questions or follow-up questions, we’re always here to help! Just be sure to make a new post to safegraphdata, as we aren’t monitoring old threads at this time. Thanks!
