Installing Charles WebProxy

How to install Charles WebProxy, trust its root certificate and enable SSL proxying so you can inspect the HTTPS traffic your scraper sends.

Downloading and installing

The steps will be similar on macOS and Linux, for simplicity we focus on Windows. Download the binary here: https://www.charlesproxy.com/download/

Trusting Root Certificate

Charles needs to have it's root certificate trusted by your computer in order to decrypt TLS connections. Click on Help -> SSL Proxying -> Install Charles Root Certificate .

A new window will open, click on Install Certificate... and select either Current User or Local Machine. This only matters if you want to use Charles on multiple users. Click Next .

You then need to select the certificate store. This is really important, as Charles needs to be in the Trusted Root Certification Authorities folder. You can place it here by selecting Place all certificates in the following store and selecting the correct folder after clicking on Browse. Then click Next and Finish, it should show like this:

Decrypting TLS

By default, Charles will not decrypt https traffic. In order for this to work you need to go to Proxy -> SSL Proxying Settings and enable Enable SSL Proxying, you must also add a wildcard to the Include section. After you're done it should look like this:

Finished!

You're all set! You can now route your script's traffic through http://127.0.0.1:8888 and have it show up in Charles. Or you can enable Proxy -> Windows Proxy to have your browser automatically route it's traffic through Charles.

Pro tip: Route your browser with a Proxy profile so that it only shows traffic in Charles when you need to.

I recommend reading their official documentation here: https://www.charlesproxy.com/documentation/ as there are much more features that I won't cover in-depth here. Get in touch: discord.gg/akamai