API Call Timeout Issue

Questions & Help
, , , ,
1

Hello there! I’m working on pulling visitor_home_cbgs for a set of placekeys that I’m interested in. I’ve been running a loop to make a request, and I’m aware that there’s 1000 requests per minute limit on API calls. It only made about 180 requests per minute, but I received a request timeout error halfway through my pull. What’s the length for a window of API requests? 2 hours? Is my method for calling the API incorrect? Any all help would be appreciated!

Here’s my code if you’ve time to review it:

Here’s the method I’m using to make API calls for the CBGs related to the placekeys:

type or paste code here

This is my loop for fetching CBGs related to my interested placekeys:

collected_pks = []
for pk in os.listdir('poll_cbgs'):
    pk = pk.split('_related_')[0]
    collected_pks.append(pk)

count = 0
for pk in tqdm.tqdm(poll['placekey'].values.tolist()):
    if pk in collected_pks or pk in null_pks:
        continue
    data_dict = {
        'placekey': [],
        'cbg': []
    }
    data, response_success = fetch_poll_cbgs(pk)
    
    if 'error' in data:
        print('Request Timeout Occurred')
        
    
    if response_success:
        if data['extensions']['row_count'] == 0:
            null_pks.append(pk)
            continue
        cbgs = data['data']['lookup']['weekly_patterns'][0]['visitor_home_cbgs']
        pk = data['data']['lookup']['weekly_patterns'][0]['placekey']
        for key in cbgs:
            data_dict['placekey'].append(pk)
            data_dict['cbg'].append(key)
        pickle.dump(data_dict, open('poll_cbgs/'+pk+'_related_cbgs.pickle', 'wb'))
        count += 1
        if count > 1000:
            print('Sleeping!')
            time.sleep(30)
            count = 0
    else:
        # print('Response Failed')
        null_pks.append(pk)
        # print(data)
2

Also, I couldn’t post my method for making API calls, here it is:

def fetch_poll_cbgs(pk):
    headers = {
        'Content-Type': 'application/json',
        'apikey': api_key
    }
    query = """
    query {
      lookup(placekey: """ + '"' + pk + '"' + """) {
        weekly_patterns (start_date: "2020-11-02", end_date: "2020-11-09") {
          placekey
          date_range_start
          date_range_end
          visitor_home_cbgs
        }
      }
    }
    """

    req = requests.post(
        'https://api.safegraph.com/v2/graphql',
        json={
            'query': query
        },
        headers=headers
    )
    try:
        return req.json(), True
    except:
        print('Request Failed')
        print(req)
        return req, False