We’ve published the restated version of Monthly Patterns but are still waiting on the Weekly Patterns data to be updated.
Here’s a list of Placekeys that have been dropped from the Advan data because of some issue with the Place. I recommend dropping these from any longitudinal analysis:
I just wanted to follow up on this. I downloaded December 2022 of Advan MP and it looks anomalous compared to other months. Half as many rows when I aggregate all the files, many fewer POIs in the data, and a total size of ~6 GB while most other months are about 20 (as an unzipped csv after I drop a bunch of columns, but either way, my process has been consistent this whole time). If you did end up refreshing the patterns data, how can we access the new files? If I order new data, nothing seems to change. Thanks so much for your help.
Hey @Andrew_Renninger_Penn our new product is launching this Wednesday with the new data loaded. Apologies for the inconvenience and thanks for your patience. You’ll get an email notification about the launch.
Excellent. Thanks.
You can try re-downloading the data on our new platform: https://app.deweydata.io/
This has the latest data from Advan, which will hopefully resolve some of the discrepancies you’re seeing.
Your community credentials should work as the log-in: Platform Updates FAQ
Hi Evan,
The new data still appears a bit “light” compared to the other months. December 2022 had 3.3GB of raw data in 14 files, while the month before and after had around 9.7 and 11GB of raw data.
Hi Evan— There is still a big drop in the new batch (December 2022): this dataset is only tracking 4 million POIs, about half the level from 2019 to 2021. Any idea what might have changed between those months?
Thanks @Martin_Andersen_UNC_Greensboro and @Andrew_Renninger_Penn, your feedback helps.
Have you tried removing the “excluded_placekeys” shared above from all of your pre-Dec 2022 data?
From my understanding, Advan started adding more checks to their data around Dec 2022 to test for and remove data because of “bad” polygons (i.e. polygons that now get denied by internal checks due to an unreasonable size/the number of vertices.). There are over 1m POI on that list which may make up for the difference.
Instead of backfilling the historic data and removing those, for now we just have the list but I can see if we can pre-remove these from our version of their data.
Hi Martin,
team dewey here - can you elaborate more on what you are seeing? would it be correct to interpret this as the number of placekey matching the exclude-list has decreased since Dec 2022?
Hi Felix, for Dec 2022 it is simply that there are fewer placekeys overall (it fell by about half–you can’t tell but that’s a log scale). I also created a separate thread on some other issues with the 2023 data and July 23, in particular.
got it. so the unique placekey that has visits has dropped at these date: Dec 2022 and July 2023?
I’m sure I can find those threads but do you mind linking to them for me?
This is my plot of average poi foot traffic at nyc, and except from the abnormal drop starting Jan 2023, I can also see another drop since Jun 2023. Is Advan correcting again about the poi?
Thanks!
Hello! We tracked the issue to some massive POI changes by SafeGraph. Technically speaking the POI changes are trending the correct way, by substantially lowering the area of some shared POIs (making them more accurate). However, due to the number of POIs sharing some polygons, a large change in even a few polygons resulted in massive cumulative changes in traffic across the board. In particular while the number of POIS is fairly stable (and as a matter of fact jump by almost 20% from Dec 2022 to June 2023) the area of these pois dropped from 17.2M m2 (total over pois) in Dec POIs to 15M m2 in March and down to 4.3M m2 in May and 4.1M m2 in June. So we have # of poi going up by 20% while area is dropping by 75% total.
We suggest that you, and/or your end users, remove the shared POIs from your calculations; you will find that the drops are not anywhere near as large. If you still see a lot of volatility, please keep only those set of POIs whose traffic (and therefore their size) has not materially changed throughout time.