Tutorial: How to Access Files via the API

Help
,
1

To test out the Data API yourself, check out this Google Colab notebook tutorial.

2 Likes
2
3

It looks like the following:

Should read:

(e.g. debian linux)

4

I am can use this API tool to download files. However, the code starts to become super slow after I downloaded 277 files. It has been 2 hours, the code always runing but no more files downloaded.

Do you know why? any suggestions on best practices of this?

Best
Yang

5

Hi @yangsong, I’m curious if this error could have been due to a network disconnection or something. A few questions:

  1. Did the downloads stop completely? Or were there still files slowly being downloaded?
  2. Did you confirm you had more than 277 files to download? The code should have stopped running after all files are done anyway.
  3. Have you tried re-running the code? If you are familiar with Python, you might be able to skip the files that are already successfully downloaded.
  4. Can you confirm the files that were downloaded look as expected? Also, specifically, does the 277th file (the final one that downloaded) look correct?
6

Thanks for the answer.

The downloads didn’t stop but keep running with no new files created for like 3hours. I stopped the python file and rerun the code. It works again. I guess I need just to rerun the code to fix this.

1 Like
7

@ryank I have another question. I wonder will the content in my api request URL change?

For example, I have a url that point to a file: /api/data/v2/data/2022/12/26/SAFEGRAPH/WP/20221226-safegraph_wp_cpgp_part29_0

If I download this URL using the python API file you provided yesterday vs today, will the content in the file change?

Best

8

I ask my colleague to help me with fetching API data.

When he tried, it shows that I am still not authorized to access data. See below

mounire@mounire-dev:~/TST$ cat .env
AMPLIFY_API_KEY=tVl6Xoew.FkvKzMKHRmQsSRwcGXF6VIGO2nhEJpuzyDyDUFxHjRT6ONats6k4P349
mounire@mounire-dev:~/TST$ echo AMPLIFY_API_KEY tVl6Xoew.FkvKzMKHRmQsSRwcGXF6VIGO2nhEJpuzyDyDUFxHjRT6ONats6k4P349 mounire@mounire-dev:~/TST pip install -r ./requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: tqdm==4.65.0 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 1)) (4.65.0)
Requirement already satisfied: python-dotenv==1.0.0 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 2)) (1.0.0)
Requirement already satisfied: requests==2.28.1 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 3)) (2.28.1)
Requirement already satisfied: click==8.1.3 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 4)) (8.1.3)
Requirement already satisfied: charset-normalizer<3,>=2 in /home/mounire/.local/lib/python3.8/site-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (2.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (2019.11.28)
DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: gpg 1.13.1-unknown has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of gpg or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
mounire@mounire-dev:~/TST$ python get_product_data.py --api_url “https://app.deweydata.io/external-api/v2/products/176f0262-c1f6-4dbe-be43-6a6eb21bcf8a/files” --download_dir ./olga
Traceback (most recent call last):
File “get_product_data.py”, line 67, in
main()
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 1130, in call
return self.main(*args, **kwargs)
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 1055, in main
rv = self.invoke(ctx)
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 760, in invoke
return __callback(*args, **kwargs)
File “get_product_data.py”, line 52, in main
raise Exception(res.json())
Exception: {‘detail’: ‘Unauthorized’}
mounire@mounire-dev:~/TST$

9

Redirecting any future responses/inquiries related to the @semukhina API error to this thread instead.