All posts

craft ·

Why Sites Work in Incognito but Not Regular Browser: Complete Developer Guide

Comprehensive troubleshooting guide for when websites load in incognito but fail in regular browsers. Covers DNS cache, HSTS, cookies, service workers, and provides step-by-step solutions for developers.

Table of Contents

  1. What: The Problem
  2. Why: Root Causes
  3. How: Solutions
  4. Prevention & Best Practices
  5. Quick Reference

What: The Problem

Symptoms

  • Website loads perfectly in incognito/private mode
  • Same website fails in regular browser with errors like:
    • DNS_PROBE_FINISHED_NXDOMAIN
    • NET::ERR_CERT_AUTHORITY_INVALID
    • ERR_TOO_MANY_REDIRECTS
    • Site stuck loading or showing old version
    • HTTPS security warnings that can’t be bypassed
    • “This site can’t be reached” errors

When This Commonly Occurs

  • After DNS changes or domain migrations
  • During development with localhost
  • After SSL/TLS certificate updates
  • When switching between HTTP and HTTPS
  • After deploying to new hosting providers (Vercel, Netlify, etc.)

Why: Root Causes

1. DNS Cache (Most Common for Domain Changes)

Browser DNS Cache

  • Browsers maintain their own DNS cache separate from system cache
  • Incognito Mode ignores DNS cache, fetching fresh data each time
  • Failed DNS lookups are cached, causing persistent NXDOMAIN errors

System DNS Cache

  • Operating system caches DNS at system level
  • Affects all browsers except incognito mode
  • Can persist even after browser restarts

2. Browser Cache & Cookies

HTTP Cache

  • Incognito Mode ignores cached data, so it loads a fresh version of the site — like hitting a reset button
  • Cached redirects (301/302) can cause loops
  • Old cached resources may conflict with new deployments

Cookies & Session Storage

  • Cookies store login info, preferences, and session data. But when these cookies get expired, broken, or just plain weird, a website might refuse to load properly
  • Corrupted cookies can break authentication
  • Session conflicts between old and new versions

3. HSTS (HTTP Strict Transport Security)

What HSTS Does

  • The HTTP Strict-Transport-Security response header (often abbreviated as HSTS) informs browsers that the host should only be accessed using HTTPS
  • Browser remembers to use HTTPS for specific duration
  • Cannot be bypassed with “proceed anyway” option

Why It Causes Issues

  • Previous HTTPS visit cached HSTS settings
  • Certificate changes or misconfigurations trigger errors
  • Unlike other HTTPS errors, HSTS-related errors cannot be bypassed

4. Browser Extensions

  • Since most browsers disable extensions in Incognito Mode, the website suddenly works fine
  • Common problematic extensions:
    • Ad blockers
    • Privacy extensions
    • CORS/security modifiers
    • Developer tools
    • VPNs/proxy extensions

5. Service Workers & PWA Cache

  • Service workers persist in regular browsing
  • Can serve outdated cached content
  • Not active in incognito mode
  • Can interfere with new deployments

How: Solutions

Quick Fix Methods (Try First)

1. Force Hard Refresh

  • Windows/Linux: Ctrl + Shift + R or Ctrl + F5
  • Mac: Cmd + Shift + R

2. Clear Chrome DNS Cache

chrome://net-internals/#dns
→ Click "Clear host cache"

chrome://net-internals/#sockets  
→ Click "Flush socket pools"

3. Clear System DNS Cache

Windows:

ipconfig /flushdns

macOS:

sudo dscacheutil -flushcache

Linux:

sudo systemd-resolve --flush-caches
# or
sudo service nscd restart

Comprehensive Solutions

Method 1: Clear All Browser Data (Nuclear Option)

Chrome/Edge:

  1. Settings → Privacy and Security → Clear browsing data
  2. Time range: “All time”
  3. Select ALL:
    • Browsing history
    • Cookies and other site data
    • Cached images and files
    • Site settings
    • Hosted app data
  4. Clear data and restart browser

Firefox:

  1. Settings → Privacy & Security → Clear Data
  2. Or: Ctrl + Shift + Del
  3. Select “Everything” for time range
  4. Check all boxes
  5. Clear and restart

Method 2: Clear HSTS Settings

Chrome/Chromium:

chrome://net-internals/#hsts
→ Enter domain in "Query HSTS/PKP domain"
→ Enter domain in "Delete domain security policies"  
→ Click Delete

Firefox:

  1. History (Ctrl + Shift + H)
  2. Find the website
  3. Right-click → “Forget About This Site”

Or manually:

  1. Navigate to about:support
  2. Open Profile Folder
  3. Close Firefox
  4. Edit SiteSecurityServiceState.txt
  5. Remove domain entry

Edge:

edge://net-internals/#hsts
→ Same as Chrome process

Method 3: Clear Site-Specific Data

Chrome (Recommended for development):

  1. Open DevTools (F12)
  2. Application tab
  3. Storage → Clear site data

Alternative:

  1. Click padlock icon in address bar
  2. Site settings
  3. Clear data

Method 4: Service Worker Cleanup

Chrome DevTools:

  1. Application tab → Service Workers
  2. Check “Update on reload”
  3. Click “Unregister” for all workers
  4. Clear storage

Or via Chrome settings:

chrome://settings/content/all
→ Find your site
→ Delete all data

Method 5: Extension Troubleshooting

  1. Disable all extensions: chrome://extensions
  2. Test the site
  3. If it works, enable extensions one by one
  4. Identify and remove/configure problematic extension

Development-Specific Solutions

For Localhost Issues:

# Clear Chrome localhost HSTS
chrome://net-internals/#hsts
→ Delete: localhost
→ Delete: 127.0.0.1

For New Deployments:

  1. Clear all site data in DevTools
  2. Unregister service workers
  3. Hard refresh: Ctrl + Shift + R
  4. Test in new incognito window first

Prevention & Best Practices

For Developers

1. DNS Change Strategy

  • Before changing DNS:

    • Lower TTL values 24-48 hours in advance
    • Document old DNS settings
    • Prepare rollback plan
  • During DNS change:

    • Keep both old and new servers running
    • Use 302 (temporary) redirects initially
    • Monitor both servers
  • After DNS change:

    • Wait for propagation (up to 48 hours)
    • Test from multiple locations/devices
    • Switch to 301 redirects only after verification

2. HSTS Implementation

# Start with short max-age for testing
Strict-Transport-Security: max-age=300

# Gradually increase after verification
Strict-Transport-Security: max-age=31536000; includeSubDomains

3. Service Worker Versioning

// Add version to service worker
const CACHE_VERSION = 'v1.2.3';

// Implement proper cache busting
self.addEventListener('activate', event => {
  event.waitUntil(
    caches.keys().then(cacheNames => {
      return Promise.all(
        cacheNames
          .filter(name => name !== CACHE_VERSION)
          .map(name => caches.delete(name))
      );
    })
  );
});

4. Development Headers

// Prevent aggressive caching during development
response.headers.set('Cache-Control', 'no-cache, no-store, must-revalidate');
response.headers.set('Pragma', 'no-cache');
response.headers.set('Expires', '0');

For Users/Testers

Regular Maintenance

  • Clear browser cache monthly
  • Update browser regularly
  • Review and clean extensions periodically

Testing New Deployments

  1. Always test in incognito first
  2. Clear site data before testing updates
  3. Document which browser/version you’re using
  4. Check from multiple networks if possible

Quick Reference

Diagnostic Commands

IssueCheck Command/URL
DNS Cachechrome://net-internals/#dns
HSTS Statuschrome://net-internals/#hsts
Service Workerschrome://serviceworker-internals
Site DataDevTools → Application → Storage
Extensionschrome://extensions
System DNS (Windows)nslookup domain.com
System DNS (Mac/Linux)dig domain.com

Emergency Fixes Priority

  1. First Try: Hard refresh (Ctrl+Shift+R)
  2. Then: Clear DNS cache (chrome://net-internals/#dns)
  3. Next: Check in different browser
  4. Then: Clear site-specific data
  5. Last Resort: Clear all browser data

Red Flags That Indicate This Issue

  • ✅ Works in incognito
  • ✅ Works in different browser
  • ✅ Works on different device
  • ✅ Started after DNS/domain change
  • ✅ Started after HTTPS implementation
  • ✅ Shows certificate/security errors

Time Estimates for Fixes

SolutionTime to ExecuteSuccess Rate
Hard Refresh5 seconds20%
Clear DNS Cache30 seconds40%
Clear Site Data1 minute60%
Clear HSTS2 minutes80%
Full Browser Reset5 minutes95%
System DNS Flush1 minute30%

Key Takeaways

  1. Incognito works because it bypasses: DNS cache, browser cache, cookies, HSTS settings, extensions, and service workers

  2. Most common cause for domain changes: DNS cache at browser level, not system level

  3. Fastest universal fix: Clear site-specific data in DevTools → Application → Clear storage

  4. Prevention is key: Implement proper cache headers, version service workers, and use gradual HSTS rollout

  5. Not a bug, it’s a feature: These caches exist to improve performance and security—they just need proper management during changes


Additional Resources


Last updated: August 2025
Save this guide for future reference when deploying or migrating domains