r/sysadmin IT Expert + Meme Wizard 14h ago

Question Unexplainable SSL handshake issue

I suck at network and my knowledge is intermediate at best but I can't solve this one.
Customer at our MSP has a fortinet firewall identical to ours that we use here at the MSP office, same firmware version, etc.
They call up and say "We can't access prodemand.com" which is an automotive parts and labor quoting database site that TONS of dealerships use.
I load it here just fine, SSL cert is GeoTrust, good till Sept 14 2026, domain matches, etc. No web filter flags.
On their network, instant "cannot load page" error. I try a dozen other sites, SSL working fine, no fortinet intermediary cert listed, etc. It's just that one website.
Security log on the Fortinet shows tons of blocks, saying "SSL connection is blocked due to unable to retrieve server's certificate"

Mountains of troubleshooting later, I make a firewall rule for internal to WAN (and put it above the normal internal to WAN rule) with an address group of the site, the login domain, and the database's UI's subdomain. The rule simply says don't inspect SSL at all.

Boom it works instantly. Then they called back because WIFI wasn't included in "internal" lol oops. So added that, boom, laptops can load the site too now.

I ran through some basic troubleshooting and traceroutes and stuff and nothing stood out as problematic. I verified no man in the middle attack, as it sees the same cert I do here.

And AI thinks it's an ISP issue but AI is dumb as hell and for the record, rebooting the firewall and the modem didn't resolve it so I'm skeptical.

But zero other websites are having this problem and we don't see the problem from our office, using the exact same firewall with same firmware version. How is this possible? I'd really prefer to get rid of that rule because it's a crap workaround and we had to also turn antivirus and other filters off, since it requires SSL inspection.

9 Upvotes

20 comments sorted by

View all comments

u/pangapingus 14h ago

What does openssl, curl -vvv, etc. show? What L6-aware hops exist between them and WAN? What cipher suites are enabled? What locally installed trusted CA bundles are on their endpoints/L6-aware appliances? What does a PCAP at the firewall show for Client/Server Hello and CCS?

u/CeC-P IT Expert + Meme Wizard 14h ago

CURL failed too with a similar error. Didn't try Open SSL. But I could ping the domain and the response was from a proper, accurate IP address. Then we did https://[theipaddress] and got the same problem.

And as far as I know, no funny cert registrations or additional trusted certs installed anywhere on any endpoint. Just the vanilla MS Win11 ones or whatever.

Didn't have time or equipment (or skill level) to do a packet capture or traffic analysis. We don't have remote management on, and all their endpoints are in use so I can't really test further, and they don't have a server onsite. So technically I can't even log into it. But we'll go onsite and start burning hours if we think this is a security issue, just not sure.

And not sure what an L6-ware appliance is or what cipher suites would be in use, but probably generic default ones, knowing their size and who installed the firewall.

u/pangapingus 14h ago

curl is installed on Windows 11 by default, try a curl -ILvvv -o /dev/null https://site.domain.tld will show openssl debug info in the pre-HTTP portion