"cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" on AWS Lambda using a layer – Python

The Problem:

When attempting to deploy an AWS Lambda function that utilizes the Requests library for web scraping, an error occurs during execution:

"Unable to import module ‘lambda_function’: cannot import name ‘DEFAULT_CIPHERS’ from ‘urllib3.util.ssl_’/opt/python/urllib3/util/ssl_.py"

Despite adding a layer to the Lambda function, which includes the necessary Requests, Pandas, and BeautifulSoup4 dependencies, the import error persists. The layer was created using Python 3.9.13, while the Lambda function’s runtime is set to Python 3.9.

The bucket and folder for S3 storage exist, and the Lambda function has the appropriate role to access S3. Troubleshooting steps taken, such as manually installing urllib3==1.26.15 in the layer, have not resolved the issue.

The objective is to successfully scrape a website and save the data to S3 using the Lambda function.

The Solutions:

Solution 1: Install required dependency versions

To resolve the "cannot import name ‘DEFAULT_CIPHERS’ from ‘urllib3.util.ssl_" error in AWS Lambda using a layer, follow these steps:

1. Install the specific versions of the required dependencies:

   • pip install requests==2.25.0 -t ./python --no-user
   • pip install beautifulsoup4 -t ./python --no-user
   • pip install pytz -t ./python --no-user

2. Download the whl files from PyPI:

   • numpy-1.24.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   • pandas-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

3. Unzip the whl files and move the extracted content to the python folder.

4. Zip the updated python folder and upload it as a layer to AWS Lambda.

5. Assign the layer to your Lambda function. This will ensure the correct versions of dependencies are available during execution.

Solution 2: Pin urllib3 to a version lower than 2.0

The error occurs because you’re using botocore, which doesn’t support urllib3 2.0. To resolve this, explicitly pin urllib3 to a version lower than 2.0 in your project using:

urllib3<2

Additionally, follow these guides to deploy your Python Lambda function using a .zip file archive or a container image:

• Deploying Python Lambda functions with .zip file archives
• Deploying Lambda functions as container images

Solution 3: Explicit Requests Version Specification

To resolve the import error, specify the exact version of the requests library when installing it within the Python layer:

!pip install requests==2.28.2 -t ./python --no-user

By specifying the version (in this case, 2.28.2), you ensure that the compatible version of the requests library is included in the layer and will be used by your Lambda function, resolving the import error.

Solution 5: Use a compatible Python version

When creating the Lambda function, use Python version 3.11. Lower Python versions may encounter this error.