Reading In a Portion of Advan Data

[A1 (A)]

Hi,

When I try to download a sample dataset from Advan (weekly patterns to be exact), the dataset is so big that I can’t open it. Is there a way to only read the first 100 rows or so, that way I can see how the data looks and practice with it?

Any step by step instructions on how to properly download and sample code on how to view a portion of the sample data will help!

[evan-barry-dewey (Evan Barry)]

We have this tutorial for taking a quick look at the data without downloading

A quick look at Advan (or SafeGraph) data without downloading

Of course, you can read and take a look at Dewey datasets directly from a URL without downloading them to your hard drive. Sometimes, I want to see the data to get a quick idea about it. I will continue using the functions that I introduced in my previous post here. I have weekly data sets looks like: [image] First, I want to look at one pattern file and here is how. import gzip from io import BytesIO import traceback # Directly read csv data from gz file def read_data(token, file_url, tim…

We’re also launching a large upgrade to our product in a few weeks that will allow you to see sample row and all of the column attributes for every dataset on the platform without downloading the data.

For now, you can find the Advan attributes here: Documentation (Public) - Google Drive

[InvTech]

This will help you to use Python code direct in R without needing to convert them to R codes:

Bulk data download and Python in R

This will be the last post for downloading data from Dewey. This post will allow users to download data for a specific period for specific data (e.g., from March 2020 to December 2022 Advan weekly patterns). Also, you can download data for a specific week. The download code is written in Python though, and I will not translate it to R code. The good news is you can use any Python codes in R without converting them to R codes. This article will help you download files. However, in general, this a…

Donn.

[A1 (A)]

Okay thanks. So all I will need to do is copy this code exactly into python with only changing the url link?

[InvTech]

Yes, that should work.

But still, the current function will read the entire one csv file (.gz), which can be ~280MB on your computer memory. I think it should be okay, but if you really want to read the first 100 or so rows, please add nrows = 100 parameter to pd.read_csv function as below.

# Directly read csv data from gz file
def read_data(token, file_url, timeout = 300):
    # token = tkn
    # file_url = "/api/data/v2/data/2022/12/26/ADVAN/WP/20221226-advan_wp_pat_part99_0"
    # timeout = 300
    src_url = DEWEY_MP_ROOT + file_url
    response = requests.get(src_url, headers={"Authorization": "Bearer " + token}, timeout=timeout)
    try:
        csv_df = pd.read_csv(BytesIO(response.content), nrows = 100, compression="gzip")
    except gzip.BadGzipFile: # not gzip file. try normal csv
        csv_df = pd.read_csv(BytesIO(response.content), nrows = 100)
    except:
        print("Could not read the data. Can only open gzip csv file or csv file.")
    return csv_df