Tool: Automated Data Access via API with Python

Update: A new guide for using the latest version of the API with Python can be found here: Bulk Downloading Data using v3 API using Python


Dewey launched a new and improved data delivery/access system on August 23, 2023. As such, the previous approaches to accessing data via API are now outdated.

This notebook acts as a tutorial and tool to help you access your data quickly in the new system.

  • The end-to-end tutorial shows each step required, from creating an API key and ordering data all the way to loading it back into a Pandas dataframe
  • The Google Colab notebook provides a free environment to run the code on, allowing you to download the data without worrying about setting up your own environment first
  • The Python code can be customized to your needs in Colab or in your own environment – this means you can filter/process the data prior to moving it to your analysis environment
  • Although written in Python, the tutorial demonstrates the general approach required for API access, meaning you can convert it to another language as you see fit
  • A similar tutorial exists for R users

Please share questions, feedback, and concerns below!

I ask my colleague to help me with fetching API data.

When he tried, it shows that I am still not authorized to access data. See below

mounire@mounire-dev:~/TST$ cat .env
AMPLIFY_API_KEY=tVl6Xoew.FkvKzMKHRmQsSRwcGXF6VIGO2nhEJpuzyDyDUFxHjRT6ONats6k4P349
mounire@mounire-dev:~/TST$ echo AMPLIFY_API_KEY tVl6Xoew.FkvKzMKHRmQsSRwcGXF6VIGO2nhEJpuzyDyDUFxHjRT6ONats6k4P349 mounire@mounire-dev:~/TST pip install -r ./requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: tqdm==4.65.0 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 1)) (4.65.0)
Requirement already satisfied: python-dotenv==1.0.0 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 2)) (1.0.0)
Requirement already satisfied: requests==2.28.1 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 3)) (2.28.1)
Requirement already satisfied: click==8.1.3 in /home/mounire/.local/lib/python3.8/site-packages (from -r ./requirements.txt (line 4)) (8.1.3)
Requirement already satisfied: charset-normalizer<3,>=2 in /home/mounire/.local/lib/python3.8/site-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (2.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.28.1->-r ./requirements.txt (line 3)) (2019.11.28)
DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: gpg 1.13.1-unknown has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of gpg or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
mounire@mounire-dev:~/TST$ python get_product_data.py --api_url “https://app.deweydata.io/external-api/v2/products/176f0262-c1f6-4dbe-be43-6a6eb21bcf8a/files” --download_dir ./olga
Traceback (most recent call last):
File “get_product_data.py”, line 67, in
main()
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 1130, in call
return self.main(*args, **kwargs)
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 1055, in main
rv = self.invoke(ctx)
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/mounire/.local/lib/python3.8/site-packages/click/core.py”, line 760, in invoke
return __callback(*args, **kwargs)
File “get_product_data.py”, line 52, in main
raise Exception(res.json())
Exception: {‘detail’: ‘Unauthorized’}
mounire@mounire-dev:~/TST$

Hi @semukhina , it looks like this would be an authorization issue with the API key. It’s possible the API key is not being saved after it’s generated – please see this post. You would need to create a new API key, save it, then try accessing the data again.

Also, it is recommended to avoid including your API key in any posts/messages. @JosephZ it’s possible the above API key was not saved anyway, but could you make sure to expire it if needed?