Ryan: I have grabbing the data from the CLI interface, not S3. Can you please advise?
@Bruce_Mizrach_Rutgers_University can you elaborate what you need help with or what you do not understand?
These are the paths you use with the CLI
If you are able to wait a couple of days, we are going to make this backfill better organized and easier for you to work with, and we can provide more documentation.
Here is how I connect. The web interface abstracts from these S3 paths
@Bruce_Mizrach_Rutgers_University OK. we are actively updating this, so the easiest thing may be to just give us 24 hours and look for an announcement in #announcements regarding the backfill, and that should make everything clear. I apologize for the inconvenience.
Can you explain the backfill that is already there?
@Bruce_Mizrach_Rutgers_University I can try to explain. Can you tell me how long have you been working with SafeGraph data, and been in the Slack group?
We re-do backfills ~ 2x per year. The previous backfill was done in May 2020 (not sure if you were in teh community at that time). The most recent backfill was done a few weeks ago at start of Dec 2020.
Currently the data from those two different backfills is poorly labeled in our catalog, so we are fixing that and trying to make things clear. Our recommendation will be to use the data from Dec 2020 backfill and ignore data from previous backfills, which we considered deprecated, but right now it is not clear in the catalog which is which. I tried to explain earlier in this thread, but if that doesn’t make sense, then just give us a little time to fix the catalog.
Since April 2020. I think I started pulling data after the May 2020 backfill
Can you explain the revision process? Economists really care about vintages of data. GDP gets revised many times, but the Fed makes decisions based on the first estimate.
@Bruce_Mizrach_Rutgers_University I can refer you to the following resources that try to provide more context on revisions/ backfills/ updates.
- Release notes for the Dec 2020 release, which explain all the major new features applying to the Dec 2020 data.
- General info about versioning and backfills: FAQs | SafeGraph Docs
Broadly speaking, the goal of the backfill is to provide you a longitudinal view of foot-traffic that is comparable across time (i.e., measured, computed, aggregated, etc all using the same methods and algorithms).
will the number of visits to the Starbucks in Westfield, NJ ((there is only one) for a given week change in the revision?
it’s certainly possible. Reasons that this might change include:
• we’ve changed our algorithm/sensitivity for how we measure/detect a visit to this location;
• we’ve improved our data about this point-of-interest such as its geometry, open hours, that lead to changes in visit attribution given the same GPS data.
• We’ve improved our data about an adjacent POI such that visits that were previously incorrectly attributed to the neighbor are now correctly attributed to the Starbucks (or vice versa), etc.
• other methodology changes
But the point is that these changes will be applied to ALL TIME in the backfill so although the measurements for a given week may change between one backfill vs the next backfill, the visits within a backfill across weeks are consistent in terms of methodology, etc.
I believe our plan is to keep the data from the previous backfill available to you for legacy analysis so none of the data we’ve previously shared should disappear. We just have new data that we think is better that will be listed in a new/different path/download.
Sorry about this, guys. I’ve cleared up the stores and posted an update here with actions you can take: Workspace Deleted | Slack
The web interface for the backfill is not properly configured.
Sorry about that. Let me check.
There we go, sorry about that.
I am concerned that in the revised data the total devices seen has gone done. What is the cause of this?
done = down
Just to check, Bruce, you mean in the Backfill and Weeklies for Dec? Let me tag @Ryan_Fox_Squire_SafeGraph in for that one.