While reversing a web application, I uncovered an interesting API endpoint and decided to access it via curl only to be blocked by a WAF despite accessing it freely via my browser. Tried the same request using python's requests library and got a similar WAF block.
After making sure I wasn't missing any WAF cookies or request headers I tried a few other methods that included:
- Java's HttpURLConection, Python's Requests (Blocked by WAF)
- Tunnel curl request through burpsuite proxy (Worked)
- Tunnel request through a simple socketserver python proxy (Blocked by WAF)
- Tunnel request through mitmproxy(Surprisingly also worked)
The results had me curious so I researched rules WAFs use to spot scraper bots and that led me to JA3 Fingerprint - A method for profiling clients based on the structure of the "Client Hello" sent by the client at the start of a TLS handshake between client and server. The structure is md5summed to create a unique 32 character id for different browser versions.
Checking the "Client Hello" packet of a curl request vs browser (Firefox) request using wireshark immediately reveals obvious differences especially in the cipher suites. Firefox sends 17 ciphers while curl sends 31 one of which includes Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff) a cipher suite automatically added by OpenSSL - the cryptography library curl and python requests use. With no major browser using OpenSSL, it was an easy way for WAFs to spot non-browser requests.
While mitmproxy provided me a solution, it felt like doing too much so I used a bit of google-fu and discovered the game changing Curl-impersonate , while still in its infancy it perfectly mimics browsers (Firefox and Chrome at the moment of writing) by using nss rather than OpenSSL amongst other things.
WAFs are evolving and I expect to see more adoption of TLS Fingerprinting, staying one step ahead is important.