Hello! To preface, I am VERY new to SafeGraph and am just trying to download the Core and Patterns data. I was attempting to use the one click download, however, after waiting 3 hours I still had zero percent progress. I have no problem waiting for the download, I just want to make sure that the download is actually happening. tl;dr: is it normal for datasets to take MANY hours to download (the record size is 420 million ftr.)?
Hello, @Michael_Sparks_Christopher_Newport_University Be patient and don’t refresh the page. 420 million records is a lot of data and I believe it may be taking some time to compile.
@Michael_Sparks_Christopher_Newport_University can I ask what you are working on that requires 420m rows of SG data? As pranav alluded too, 420m is a lot and normally our users end up putting filters in place so it doesn’t take forever to download
@Saumil_Shah_SafeGraph basically I just need state aggregated data on number ov visits to different types of stores by day or by week. I.e: Nevada, Convenience Stores, X,XXX,XXX visitors. I am very new to this and do not know a way to filter out the unwanted parameters out of data sets (for example I don’t need unique visitors, or distance to visitors homes). All I know is that I need bits from the Core dataset (the top and sub categories) as well as the number of visitors from the patterns dataset.
Hey Michael, in our your Shop you have the option to filter by location, brand, or industry. So for your use case you can put Nevada in the location filter and Convenience stores in the category filter
Samuil, I saw that, however that was just an example. I need that type of data for every top_category for every state in the US. Would it be easier to just go state by state, category by category?
probably not, if you need all the details for every brand/category and by location then what you did is the right format