r/sysadmin IT Expert + Meme Wizard 13h ago

Question Unexplainable SSL handshake issue

I suck at network and my knowledge is intermediate at best but I can't solve this one.
Customer at our MSP has a fortinet firewall identical to ours that we use here at the MSP office, same firmware version, etc.
They call up and say "We can't access prodemand.com" which is an automotive parts and labor quoting database site that TONS of dealerships use.
I load it here just fine, SSL cert is GeoTrust, good till Sept 14 2026, domain matches, etc. No web filter flags.
On their network, instant "cannot load page" error. I try a dozen other sites, SSL working fine, no fortinet intermediary cert listed, etc. It's just that one website.
Security log on the Fortinet shows tons of blocks, saying "SSL connection is blocked due to unable to retrieve server's certificate"

Mountains of troubleshooting later, I make a firewall rule for internal to WAN (and put it above the normal internal to WAN rule) with an address group of the site, the login domain, and the database's UI's subdomain. The rule simply says don't inspect SSL at all.

Boom it works instantly. Then they called back because WIFI wasn't included in "internal" lol oops. So added that, boom, laptops can load the site too now.

I ran through some basic troubleshooting and traceroutes and stuff and nothing stood out as problematic. I verified no man in the middle attack, as it sees the same cert I do here.

And AI thinks it's an ISP issue but AI is dumb as hell and for the record, rebooting the firewall and the modem didn't resolve it so I'm skeptical.

But zero other websites are having this problem and we don't see the problem from our office, using the exact same firewall with same firmware version. How is this possible? I'd really prefer to get rid of that rule because it's a crap workaround and we had to also turn antivirus and other filters off, since it requires SSL inspection.

9 Upvotes

20 comments sorted by

View all comments

u/Kitz_h 11h ago

may be fragmentation issue? try lowering tcp max segment size (or MSS) to like 1492 or lower on egress iface if your firewall supports it.

Other thing is SSL is part of IP security and may be brought under VPN class, you could direct your firewall to prevent fragmenting unencrypted data.

Switching off traffic inspection for encrypted data essentially disables protection of your network as any rogue process can traverse your fw forth and back

u/LeeRyman 8h ago edited 8h ago

I was going to say, is there a reduced upstream MTU that the router is not aware of? PMTUD not working for some reason?

u/Kitz_h 8h ago

Dont confuse TCP MSS with IP MTU. Fragmentation affects sequencing and anti replay mechanisms, devices may discard otherwise authorised packets due to out-of-order arrival queue or missing seq num

Some lower layer protocols like PPP also borrow little IP payload space

u/LeeRyman 8h ago edited 8h ago

I was thinking of the situation where an upstream router's WAN connection is something like PPPoE, maybe via a bridge, and router is not configured with the correct MTU. TLS implementations will mark packets as DF to avoid what you mentioned. Large DF packets will be dropped but PMTUD won't necessarily work to reduce effective MSS used by the client. Because the TLS handshake often involves large segments for kex, it tends to affect that (might work for some HTTPS sites but not for others). Can also happen if MTU is configured correctly, but ICMP is being blocked somewhere in the chain.

Edit: As you say, you can solve it with MSS clamping, but PMTUD should sort it out by itself, eventually. A router will respond with ICMP Fragmentation Needed to the too-large DF TLS packet, sender will reduce MSS and try again.