security headers – HackerTarget.com https://hackertarget.com Security Vulnerability Scanners and Assessments Tue, 25 Feb 2020 05:05:46 +0000 en-US hourly 1 https://wordpress.org/?v=5.5.12 500K HTTP Headers https://hackertarget.com/500k-http-headers/ Sun, 11 May 2014 02:52:38 +0000 http://hackertarget.com/?p=6315 Recently we crawled the Top 500K sites (as ranked by Alexa). Following requests from readers we are making available the HTTP Headers for research purposes. Download Headers (75MB) The publication of the statistics of WordPress usage is an example of the research that can be conducted. It is possible to determine Web Applications, Web Servers, […]

The post 500K HTTP Headers appeared first on HackerTarget.com.

]]> Recently we crawled the Top 500K sites (as ranked by Alexa). Following requests from readers we are making available the HTTP Headers for research purposes.

The publication of the statistics of WordPress usage is an example of the research that can be conducted. It is possible to determine Web Applications, Web Servers, Server side scripting, Load balancers and much more.

HTTP Headers that could be examined:

Security Headers
  • HTTP Only (Set-Cookie)
  • X-Frame-Options
  • X-XSS-Protection
  • X-Content-Security-Policy
Server Headers
  • Server:
  • X-Powered-By:

Recommended Tools for Analysis

A number of basic text manipulation tools will make it easier to search through the data.

Start with a *nix based system. grep, cut, sed and some simple bash scripting will make your life easier.

The file contains 5 folders with 100K headers in each. The headers will have to be correlated with the site list file to determine the site host name.

When counting sites with grep be sure to use the -m 1 this will ensure that you do not get a count of two from sites with multiple headers (HTTP 302 Redirects).

Have you seen our other Free IP and Network Testing tools.

Discover. Explore. Learn.

I want to do more recon

Next level testing with advanced Security Vulnerability Scanners.

Trusted tools. Hosted for easy access.

Use cases & more info

The post 500K HTTP Headers appeared first on HackerTarget.com.

]]>