Statistics of Common Crawl Monthly Archives

Number of pages, distribution of top-level domains, crawl overlaps, etc. - basic metrics about Common Crawl Monthly Crawl Archives
Latest crawl: CC-MAIN-2024-30

View the Project on GitHub

MIME Types

The crawled content is dominated by HTML pages and contains only a small percentage of other document formats. The tables show the percentage of the top 100 media or MIME types of the latest monthly crawls.

While the first table is based the Content-Type HTTP header, the second uses the MIME type detected by Apache Tika based on the actual content. The underlying data including page counts is provided in mimetypes.csv resp. mimetypes_detected.csv.

crawl CC-MAIN-2024-22 CC-MAIN-2024-26 CC-MAIN-2024-30
mimetype
<other> 2.9698 1.5670 0.1215
application/atom+xml 0.0998 0.1157 0.1028
application/download 0.0034 0.0026 0.0032
application/epub+zip 0.0017 0.0016 0.0017
application/force-download 0.0063 0.0065 0.0075
application/javascript 0.0009 0.0008 0.0002
application/json 0.0246 0.0268 0.0289
application/ld+json 0.0018 0.0021 0.0020
application/marc 0.0005 0.0004 0.0006
application/msword 0.0030 0.0026 0.0030
application/octet-stream 0.0404 0.0461 0.0493
application/octetstream 0.0003 0.0001 0.0002
application/pdf 0.7579 0.6999 0.9277
application/pgp-encrypted 0.0001 0.0001 0.0001
application/pgp-signature 0.0033 0.0028 0.0037
application/postscript 0.0002 0.0002 0.0002
application/rdf+xml 0.0047 0.0052 0.0057
application/rss+xml 0.0541 0.0573 0.0581
application/rtf 0.0012 0.0014 0.0019
application/save-to-disk 0.0000 0.0000 0.0000
application/text 0.0002 0.0002 0.0002
application/unknown 0.0002 0.0002 0.0004
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0025 0.0022 0.0030
application/vnd.google-earth.kmz 0.0015 0.0004 0.0008
application/vnd.ms-excel 0.0019 0.0018 0.0023
application/vnd.ms-powerpoint 0.0003 0.0002 0.0003
application/vnd.ms-word 0.0003 0.0002 0.0003
application/vnd.oasis.opendocument.text 0.0006 0.0007 0.0007
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0002 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0013 0.0014 0.0016
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0022 0.0022 0.0025
application/vnd.wap.xhtml+xml 0.0008 0.0007 0.0006
application/x-bibtex 0.0050 0.0053 0.0061
application/x-bittorrent 0.0001 0.0001 0.0002
application/x-bzip2 0.0001 0.0002 0.0001
application/x-debian-package 0.0000 0.0000 0.0000
application/x-download 0.0018 0.0018 0.0021
application/x-endnote-refer 0.0010 0.0011 0.0012
application/x-gzip 0.0004 0.0005 0.0003
application/x-httpd-php 0.0003 0.0005 0.0005
application/x-java-jnlp-file 0.0003 0.0001 0.0001
application/x-javascript 0.0001 0.0001 0.0001
application/x-json 0.0000 0.0000 0.0000
application/x-mobipocket-ebook 0.0006 0.0006 0.0005
application/x-msdownload 0.0002 0.0003 0.0002
application/x-netcdf 0.0005 0.0002 0.0002
application/x-rar-compressed 0.0000 0.0000 0.0000
application/x-research-info-systems 0.0095 0.0095 0.0105
application/x-shockwave-flash 0.0003 0.0002 0.0002
application/x-tar 0.0001 0.0002 0.0001
application/x-tex 0.0003 0.0002 0.0002
application/x-troff-man 0.0006 0.0008 0.0006
application/x-zip-compressed 0.0000 0.0000 0.0000
application/xhtml+xml 0.0097 0.0096 0.0088
application/xml 0.0239 0.0236 0.0644
application/zip 0.0002 0.0002 0.0002
audio/mp3 0.0000 0.0000 0.0000
audio/mpeg 0.0000 0.0000 0.0001
audio/x-mpegurl 0.0005 0.0005 0.0008
audio/x-pn-realaudio 0.0000 0.0000 0.0000
audio/x-scpls 0.0001 0.0001 0.0001
audio/x-wav 0.0000 0.0000 0.0000
binary/octet-stream 0.0005 0.0005 0.0006
image/gif 0.0006 0.0006 0.0007
image/jp2 0.0000 0.0000 0.0000
image/jpeg 0.0003 0.0003 0.0004
image/jpg 0.0000 0.0000 0.0000
image/pjpeg 0.0000 0.0000 0.0000
image/png 0.0002 0.0001 0.0001
image/svg+xml 0.0000 0.0000 0.0000
image/tiff 0.0000 0.0000 0.0000
image/vnd.djvu 0.0000 0.0000 0.0000
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0002 0.0003 0.0002
text/calendar 0.0279 0.0299 0.0309
text/css 0.0008 0.0008 0.0008
text/csv 0.0034 0.0037 0.0039
text/directory 0.0002 0.0002 0.0003
text/enriched 0.0001 0.0001 0.0001
text/html 95.7924 97.2201 98.3790
text/javascript 0.0002 0.0002 0.0002
text/pdf 0.0000 0.0000 0.0000
text/plain 0.0634 0.0651 0.0709
text/prs.lines.tag 0.0034 0.0044 0.0054
text/tab-separated-values 0.0002 0.0002 0.0003
text/turtle 0.0017 0.0019 0.0021
text/vcard 0.0010 0.0009 0.0009
text/x-bibtex 0.0006 0.0006 0.0006
text/x-c 0.0002 0.0002 0.0002
text/x-csrc 0.0004 0.0003 0.0003
text/x-diff 0.0004 0.0004 0.0011
text/x-patch 0.0003 0.0004 0.0003
text/x-perl 0.0001 0.0001 0.0001
text/x-vcalendar 0.0004 0.0004 0.0004
text/x-vcard 0.0018 0.0019 0.0021
text/xml 0.0575 0.0609 0.0689
unknown/unknown 0.0000 0.0000 0.0001
video/mp4 0.0000 0.0000 0.0000
video/webm 0.0000 0.0000 0.0000
video/x-ms-asf 0.0001 0.0001 0.0002

crawl CC-MAIN-2024-22 CC-MAIN-2024-26 CC-MAIN-2024-30
mimetype_detected
<other> 0.0126 0.0130 0.0149
application/atom+xml 0.1012 0.1170 0.1045
application/epub+zip 0.0021 0.0019 0.0020
application/gpx+xml 0.0005 0.0005 0.0005
application/gzip 0.0000 0.0000 0.0000
application/javascript 0.0011 0.0010 0.0004
application/json 0.0253 0.0273 0.0298
application/marc 0.0013 0.0012 0.0014
application/mbox 0.0018 0.0017 0.0021
application/msword 0.0022 0.0019 0.0022
application/octet-stream 0.0114 0.0133 0.0159
application/pdf 0.7742 0.7149 0.9474
application/pgp-signature 0.0028 0.0024 0.0038
application/pkcs7-signature 0.0005 0.0005 0.0005
application/postscript 0.0002 0.0003 0.0003
application/rdf+xml 0.0096 0.0099 0.0104
application/rss+xml 0.0842 0.0895 0.0913
application/rtf 0.0013 0.0014 0.0017
application/text 0.0000 0.0000 0.0000
application/vnd.android.package-archive 0.0000 0.0000 0.0000
application/vnd.google-earth.kml+xml 0.0040 0.0036 0.0044
application/vnd.google-earth.kmz 0.0015 0.0004 0.0008
application/vnd.ms-excel 0.0014 0.0013 0.0017
application/vnd.ms-powerpoint 0.0003 0.0002 0.0003
application/vnd.oasis.opendocument.spreadsheet 0.0004 0.0003 0.0004
application/vnd.oasis.opendocument.text 0.0008 0.0008 0.0009
application/vnd.openxmlformats-officedocument.presentationml.presentation 0.0002 0.0002 0.0003
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 0.0013 0.0013 0.0015
application/vnd.openxmlformats-officedocument.wordprocessingml.document 0.0022 0.0022 0.0025
application/x-bibtex-text-file 0.0097 0.0098 0.0107
application/x-bittorrent 0.0002 0.0002 0.0002
application/x-bzip2 0.0000 0.0000 0.0000
application/x-dosexec NaN 0.0000 0.0000
application/x-endnote-refer 0.0014 0.0015 0.0019
application/x-mobipocket-ebook 0.0008 0.0007 0.0007
application/x-ms-asx 0.0001 0.0001 0.0002
application/x-msdownload 0.0000 0.0000 0.0000
application/x-rar-compressed 0.0000 0.0000 0.0000
application/x-research-info-systems 0.0002 0.0002 0.0002
application/x-sh 0.0012 0.0009 0.0011
application/x-shockwave-flash 0.0003 0.0002 0.0003
application/x-stata-do 0.0005 0.0004 0.0005
application/x-tex 0.0004 0.0002 0.0003
application/x-tika-msoffice 0.0023 0.0021 0.0025
application/x-tika-ooxml 0.0027 0.0023 0.0032
application/x-wais-source 0.0002 0.0002 0.0002
application/x-xz 0.0000 0.0000 0.0000
application/xhtml+xml 9.1027 9.3340 9.3604
application/xml 0.0583 0.0589 0.1062
application/zip 0.0001 0.0001 0.0000
application/zlib 0.0002 0.0002 0.0002
audio/midi NaN NaN 0.0000
audio/mpeg 0.0000 0.0000 0.0000
audio/vnd.wave 0.0000 0.0000 0.0000
audio/x-mpegurl 0.0000 0.0000 0.0000
image/gif 0.0000 0.0000 0.0000
image/jpeg 0.0002 0.0002 0.0002
image/png 0.0000 0.0000 0.0000
image/svg+xml 0.0000 0.0000 0.0000
image/tiff 0.0000 0.0000 0.0000
image/vnd.djvu 0.0000 NaN 0.0000
image/webp 0.0000 0.0000 0.0000
message/rfc822 0.0006 0.0006 0.0009
text/asp 0.0000 0.0000 0.0000
text/aspdotnet 0.0000 0.0000 0.0000
text/calendar 0.0328 0.0352 0.0367
text/css 0.0007 0.0008 0.0008
text/csv 0.0034 0.0037 0.0039
text/html 89.6012 89.3983 89.0760
text/plain 0.1067 0.1105 0.1169
text/prs.lines.tag 0.0079 0.0081 0.0106
text/tab-separated-values 0.0003 0.0003 0.0003
text/troff 0.0007 0.0009 0.0007
text/turtle 0.0017 0.0019 0.0021
text/vtt 0.0008 0.0007 0.0008
text/x-c++src 0.0002 0.0002 0.0002
text/x-chdr 0.0005 0.0006 0.0005
text/x-csrc 0.0010 0.0010 0.0009
text/x-diff 0.0012 0.0014 0.0019
text/x-jsp 0.0001 0.0001 0.0001
text/x-log 0.0020 0.0033 0.0026
text/x-matlab 0.0022 0.0023 0.0023
text/x-perl 0.0018 0.0016 0.0019
text/x-php 0.0034 0.0037 0.0036
text/x-python 0.0004 0.0003 0.0004
text/x-vcalendar 0.0005 0.0005 0.0005
text/x-vcard 0.0033 0.0033 0.0037
text/x-web-markdown 0.0004 0.0004 0.0004
text/x-yaml 0.0003 0.0004 0.0004
video/mp4 0.0000 0.0000 0.0000
video/quicktime NaN NaN 0.0000
video/webm 0.0000 0.0000 0.0000
video/x-m4v 0.0000 NaN NaN