Why are there many entries without 168 values in the VISITS_BY_EACH_HOUR column in the Weekly Patterns - Foot Traffic dataset?

I was looking at the variable VISITS_BY_EACH_HOUR in the “Weekly Patterns - Foot Traffic” dataset and it says in the documentation that there should be “the number of visits to the POI for each of the 168 hours of the week, starting at midnight on date_range_start”, but I checked for the first months of 2019 and a lot of entries have less than 168 values in that variable. Why is that? And how can it be fixed? There are also quite some entries with no information there at all.

Hi @ariadnaaz13 - can you share some examples in this thread of rows with less than 168 values in that cell? From the ones I’ve spot checked, I see 168 values even though there are blanks.

Based on this thread, it doesn’t seem like the data starts with a partial week but that’s something to consider.

I believe this dataset is appended to Places so there may be a number of rows without visit information. Advan doesn’t provide visit data on every available POI.

Hi @evan-barry-dewey , thanks for your quick reply! Some examples of rows with less than 168 values are: 223-222@5z5-3p4-35z, 22j-223@5z5-3qr-b49, 22s-225@63r-b6b-grk, and 238-223@63r-bbc-nbk.

What did you mean by “there are blanks” though?

Thanks for sharing these, we’ll review and report this back to the provider if necessary. Is this a common occurrence across POI of interest?

By blanks, I meant values = “0”. Ex.

Screenshot 2023-09-21 at 8.05.14 AM

Thank you. Please update me when you have more information about this issue. Yes, it is very common. Usually, around half of the data for one week has this issue (for the cities I selected at least, but probably it happens with all other cities). I am looking at Baltimore, Chicago, Los Angeles, Philadelphia, and San Francisco.

Ah yes, that just indicates no visits, that is fine.

@ariadnaaz13 thanks again for flagging this. We were able to confirm there are values missing in our own analysis. It seems to improve in the more recent years of data, but we’ll connect with Advan and see if this is something that can be addressed.

We’re in the process of restating the Weekly Patterns data. This should fix the issue. The updated data will be live in the next few days.

Hi @evan-barry-dewey, thank you for looking into this! Has the data been updated already? If so, is this version supposed to be complete or should I expect some more updates?

The Weekly Patterns data is currently Refreshing. Given the size, it should be ready by the start of next week.

Hey @evan-barry-dewey , I downloaded the newest version of the dataset and everything seems correct except for week 44 of 2022, where this variable has 144 entries instead of 168 (for all POIs). Can you have a look at this issue and let me know when it’s fixed? Thank you.

Can you confirm that Week 44 is the week starting October 31st?

Correct! From the 31st of October to the 7th of November.

Thanks for flagging. We were able to confirm we’re seeing the same thing in the data and have reached out to the partner to see if we can get that week restated. I’ll report back if I hear more.

Thank you! Let me know as soon as you hear back from them. It’s really important for my research to have the complete data for 2022.

@evan-barry-dewey Actually, I checked the version of the dataset that I downloaded in September and it seems that some POIs used to have the full 168 entries for the 44th week of 2022. Hence all POIs having 144 entries now for that week must be due to a mistake that was done during this last update. Have you heard back from Advan to get it fixed?

@ariadnaaz13 thanks for flagging. We heard back from the team and it sounds like the initial issue was caused by Daylight Savings Time, which removed a day from the data.

They’re going to restate the data shortly. That being said, this means we have to do a complete refresh of the datasets, which usually takes a few days.

Happy to hear that the issue is going to be fixed. Thank you!

Could you please let me know when the dataset has finished refreshing? Just to be 100% I’m downloading the correct version.

Hey @evan-barry-dewey , has the dataset finished refreshing with the correct values for this variable?

@ariadnaaz13 yes. The lower counts in the week of 2022-10-31 was due to the DST change on that week, so it was missing Sunday. This is now fixed (let me know if you see otherwise).

Great, thanks for letting me know!