Data Model

The data in ShadowTrackr is organized into indexes. Here is an overview of the indexes and what they contain.

The Assets index

This is a combined index with all hosts and urls. It exists to make searching all assets for software or cloud providers easier.

Field	Type	Description
asset	String	An IP address or url.
cloud	Boolean	True if the asset is hosted at a cloud or CDN provider. False if not.
cloudprovider	String	The name of the cloud provider.
first_seen	Datetime	The first time we saw this asset.
last_seen	Datetime	The last time we saw this asset.
remote_login_services	Boolean	True if you can use this asset to remotely login from the internet.
software	List of Strings	A list of software running on the asset.
tags	List of Strings	The ShadowTrackr tags for the asset.
warning	Boolean	True if we have a warning for this asset.
problem	Boolean	True if we have a problem for this asset.
problems	String	A description of the problems for this asset.
warnings	String	A description of the warnings for this asset.
san	List of Strings	The Subject Alternative Names (SAN) for the TLS certificate found on this host.
scanning_node	String	The node that scanned this host.

The Certificates index

This index contains all your current and past certificates (starting from your onboarding date). A certificate can be used on more than one IP address, you'll likely find the same certificate with different IP addreses. Some of the checks are done on the server's TLS configuration, so you can even see different security grades for the same certificate.

Field	Type	Description
url	String	The URL of the certificate.
ip	String	The IP address of the server the certificate is used on.
grade	String	The security grade of the certificate, we use the SSL Labs scoring mechanism.
servername	String	The servername found in the HTTP headers
not_after	Datetime	The don't use not after date of the certificate.
not_before	Datetime	The don't use before date of the certificate.
revoked	Boolean	True if the certificate is revoked. False otherwise.
signature_algorithm	String	The signature algorithm used in the certificate.
key_algorithm	String	The key algorithm used in the certificate.
sha1hash	String	The SHA1 hash of the certificate.
sha256hash	String	The SHA256 hash of the certificate.
suites	List of Strings	The cipher suites used in the certificate.
protocols	List of Strings	The protocols used in the certificate.
dh_groups	List of Strings	The Diffie-Hellman groups used in the certificate.
ecdhe_curves	List of Strings	The ECDHE curves used in the certificate.
tls12_sig_algs	List of Strings	The TLS 1.2 signature algorithms used in the certificate.
tls13_sig_algs	List of Strings	The TLS 1.3 signature algorithms used in the certificate.
subject	String	The subject (most often the url) of the certificate.
cn_without_sni	String	The common name of the certificate without the SNI extension.
cn	String	The common name in the certificate.
san	String	The Subject Alternative Name (SAN, the other urls) in the certificate.
ou	String	The organizational unit mentioned in the certificate.
o	String	The organization in the certificate.
l	String	The locality mentioned in the certificate.
st	String	The state mentioned in the certificate.
c	String	The country mentioned in the certificate.
issuer	String	The issuing authority mentioned in the certificate.
issuersubject	String	The (unparsed) issuing authority subject mentioned in the certificate.
issuer_cn	String	The common name of the issuing authority mentioned in the certificate.
issuer_o	String	The organization of the issuing authority mentioned in the certificate.
issuer_l	String	The locality of the issuing authority mentioned in the certificate.
issuer_st	String	The state of the issuing authority mentioned in the certificate.
issuer_c	String	The country of the issuing authority mentioned in the certificate.
issuer_ou	String	The organizational unit of the issuing authority mentioned in the certificate.
first_seen	Datetime	The first time we saw this certificate.
last_seen	Datetime	The last time we saw this certificate.
serial	String	The serial number of the certificate.
port	Integer	The port the certificate was found on.
cloud	Boolean	True if the certificate is found on a cloud or CDN asset. False otherwise.
cloudprovider	String	The cloud provider of the certificate.
tags	List of Strings	The ShadowTrackr tags for the certificate (often inherited from the URL).
caa_email	List of Strings	The CAA email addresses associated with the certificate.
caa_issuers	List of Strings	The CAA issuing authorities associated with the certificate.
warning	Boolean	True if the certificate has a warning.
problem	Boolean	True if the certificate has a problem.
problems	String	A description of problems for the certificate.
warnings	String	A description of warnings for the certificate.
vulnerabilities	List of Objects	A list of vulnerabilities associated with the certificate.
software	List of Strings	A list of software associated with the certificate.
grade_cap_reasons	List of Strings	Reasons why the certificate has a lower score
grade_trust_ignored	String	The grade of the certificate if trust issues are ignored.
days	Integer	The number of days until the certificate expires.
pinsha256	String	The SHA256 pin of the certificate.
renewed	Boolean	True if we found a renewed version of the certificate on all assets where it was used.
partially_renewed	Boolean	True if we found a partially renewed version of the certificate on some but not all assets where it was used.
wildcard	Boolean	True if the certificate is a wildcard certificate.
cipherorder_tlsv1	List of Strings	A list of TLSv1 cipher suites used in the certificate (in order).
cipherorder_tlsv1_1	List of Strings	A list of TLSv1.1 cipher suites used in the certificate (in order).
cipherorder_tlsv1_2	List of Strings	A list of TLSv1.2 cipher suites used in the certificate (in order).
cipherorder_tlsv1_3	List of Strings	A list of TLSv1.3 cipher suites used in the certificate (in order).
cipherorder_sslv3	List of Strings	A list of SSLv3 cipher suites used in the certificate (in order).

The Cves_assets index

This index splits all all vulnerabilities per asset. So if an asset has 3 vulnerabilities, it will appear 3 times with 3 different CVE numbers. It also shows first and last seen dates, so you can use it to check if you were vulnerable for a specific CVE two weeks ago. This can be quite handy for incident response.

Field	Type	Description
asset	String	The IP address or URL of the asset.
cve	String	The CVE number for the vulnerability.
cvss_score	String	The CVSS score for the vulnerability.
cvss_severity	String	The CVSS severity for the vulnerability.
software	String	The software that has the vulnerability.
exploited	Boolean	True if the CVE is known to be exploited in the wild.
cisa	Boolean	True if the CVE is on the CISA Known Exploited Vulnerabilities list.
first_seen	Datetime	The first time we saw this vulnerability on this asset.
last_seen	Datetime	The last time we saw this vulnerability on this asset.

The Dns index

This index contains all DNS records we found for your assets. It includes all sorts of records, like MX, A, CNAME, TXT, NS, SRC, AAAA, etc. There are subtypes for some TXT records, like SPF, DKIM and DMARC.

Field	Type	Description
adkim	String	The DMARC ADKIM tag of the record.
aspf	String	The DMARC ASPF tag of the record.
cloud	Boolean	True if the record is found on a cloud or CDN asset. False otherwise.
cloudprovider	String	The cloud provider of the record.
email	String	The email address found in the record (email field in SOA, ruf=mailto: in DKIM, etc).
expire	Integer	The expire time of the SOA record.
flags	Integer	The flags of the record (CAA, DNSKEY, etc).
first_seen	Datetime	The first time we saw this record.
last_seen	Datetime	The last time we saw this record.
k	String	The key type of the DKIM record.
mailserver	String	The mailserver of the (MX) record with the trailing dot stripped from it.
minimum	Integer	The minimum TTL of the (SOA) record.
mname	String	The master (primary DNS server) for the SOA record.
ns	String	The primary name server for the (SOA) record with the trailing dot stripped from it.
p	String	The public key for the DKIM record.
pct	String	The percentage of messages that should be subject to the DMARC policy.
port	Integer	The port of the (SRV) record.
priority	Integer	The priority of the record (MX, SRV, etc).
refresh	Integer	The refresh time of the SOA record.
retry	Integer	The retry time of the SOA record.
rname	String	The name of the responsible person for the SOA record.
rrdata	String	The raw, unparsed data of the DNS record.
rrtype	String	The type of the record (SOA, MX, etc).
rrsubtype	String	The subtype of the record (SPF, DKIM, etc).
rua	String	The RUA (report-uri) of the DKIM record.
ruf	String	The RUF (report-to) of the DKIM record.
serial	Integer	The serial number of the (SOA) record.
sp	String	The SP (selector) of the DKIM record.
t	String	The testing value of the DKIM record (y means testing).
tag	String	The tag of the (SOA) record
tags	List of Strings	The ShadowTrackr tags for the DNS record (often inherited from the URL)
target	Integer	The target of the record (CNAME, DNAME, SRV, etc).
url	String	The URL for which we found the DNS record.
v	String	The version of the DKIM record.
value	String	The value of the (CAA) record.
weight	Integer	The weight of the (SRV) record.

The Domains index

This index contains all your domains. It includes all the information we found about them, like the registrar, expiration date, DNSSEC, nameservers, etc. Both the raw WHOIS and raw RDAP data is included and searchable too.

Field	Type	Description
creation_date	Datetime	The creation date of the domain (when it was registered).
expiration_date	Datetime	The expiration date of the domain. Not all registrars provide this information, so it might be empty.
updated_date	Datetime	The date when the domain was last updated.
first_seen	Datetime	The first time we saw this domain.
last_seen	Datetime	The last time we saw this domain.
domain	String	The domain name.
dnssec	String	The DNSSEC status of the domain.
nameservers	List of Strings	The nameservers of the domain.
warning	Boolean	True if we found a warning for this domain.
problem	Boolean	True if we found a problem for this domain.
raw_whois	String	The raw, unparsed WHOIS data for this domain.
raw_rdap	String	The raw, unparsed RDAP data for this domain.
registrar	String	The registrar of the domain.
status	String	The status of the domain (registered, expired, etc).
tags	List of Strings	The ShadowTrackr tags for the domain

The Email_addresses index

This index contains all your publicly exposed email addresses, and where we found them.

Field	Type	Description
email	String	The email address.
url	String	The URL where we found the email address.
first_seen	Datetime	The first time we saw this email address.
last_seen	Datetime	The last time we saw this email address.

The Events index

This index contains all the events we found for your assets. Events are all the things that happen to your assets, like a new host, a new service, a new vulnerability, etc. You might want to ingest it in your SIEM, check the API docs for more information.

Field	Type	Description
eid	Integer	The event ID (integer valeu).
level	Integer	The level of the event. Between 0 and 100. Above 90 is a problem, above 80 a warning.
message	String	The message.
details	String	Details about the event.
link	String	The link to the assets for which the event happened.
created	Datetime	The date and time when the event happened.

The Hosts index

This index contains all your hosts.

Field	Type	Description
ip	String	The IP address of the host, can be IPv4 or IPv6.
hostname	String	The hostname we found, if any.
reverse_dns	String	The reverse DNS name of the host, if any.
tcp_ports	list of integers	The TCP ports we found open on the host.
udp_ports	list of integers	The UDP ports we found open on the host.
ports	list of integers	Mixed TCP and UDP ports we found open on the host.
software	List of Strings	The software we found on the host.
urls	List of Strings	The URLs associated with the host.
first_seen	Datetime	The first time we saw this host.
last_seen	Datetime	The last time we saw this host.
tags	List of Strings	The ShadowTrackr tags for the host.
city	String	The city where the host is located.
region	String	The region where the host is located.
region_code	String	The region code where the host is located.
country	String	The country where the host is located.
asn	Integer	The autonomous system number (ASN) of the host subnet.
asn_name	String	The name of the ASN.
asn_country	String	The country of the ASN.
asn_registry	String	The registry for the ASN.
asn_allocated	String	The date when the ASN was allocated.
prefix	String	The prefix of the host subnet.
prefix_registry	String	The registry for the prefix.
prefix_allocated	String	The date when the prefix was allocated.
isp	String	The ISP of the host.
latitude	float	The latitude of the host.
longitude	float	The longitude of the host.
remote_login_services	Boolean	True if the host can be used to remotely login from the internet.
vulnerabilities	List of Strings	The vulnerabilities we found on the host.
warning	Boolean	True if we found a warning for this host.
problem	Boolean	True if we found a problem for this host.
warnings	List of Strings	A description of the warnings we found for this host.
problems	List of Strings	A description of the problems we found for this host.
san	List of Strings	The Subject Alternative Names (SAN) for the TLS certificate found on this host.
scanning_node	String	The node that scanned this host.
portscan_duration	Integer	The duration of the port scan in seconds.
dnsscan_duration	Integer	The duration of the DNS scan in seconds.
webscan_duration	Integer	The duration of the web scan in seconds.

The Phishy_urls index

This index contains all phishy URLs we found for your assets, but only the ones that are active. Each phishy url has a score and metadata. You can use it to block incoming phishing emails, or track which domains you might want to buy. If they are for sale you can see it in the tags.

Field	Type	Description
phishy_url	String	The phishy varation of your URL.
original_url	String	The original URL, your asset.
score	Integer	The phishy score of the URL, between 0 and 100 where 100 is the most phishy.
tags	List of Strings	The ShadowTrackr tags for the URL.
isp	String	The ISP of the phishy URL.
city	String	The city where the phishy URL is located.
country	String	The country where the phishy URL is located.
first_seen	Datetime	The first time we saw this phishy URL.
last_seen	Datetime	The last time we saw this phishy URL.
jarm	String	The JARM hash of the phishy URL.
mmh3	Integer	The MurmurHash3 hash of the phishy URL.
registrar	String	The registrar of the phishy URL.
mailservers	List of Strings	The mailservers for the phishy URL.
nameservers	List of Strings	The nameservers for the phishy URL.
spf	String	The SPF record for the phishy URL.
dmarc	String	The DMARC record for the phishy URL.
babydomain	Boolean	True if the phishy URL is a babydomain (less than 30 days old).
domain_txt_verified	Boolean	True if we found DNS verification records for the phishy url from a major party (Google, Meta, Cisco, etc).
https_redirected_to_org_url	Boolean	True if the phishy URL redirects to the original URL.
whois_nameservers_not_used	Boolean	The if the nameservers from te WHOIS data are not actually used by the domain.
whois_no_dnssec	Boolean	The if the phishy URL is not using DNSSEC.

The Software index

This index tracks all the software we found on your assets. You can also use it to check which software you where running a month ago (handy for incident response).

Field	Type	Description
vendor	String	The vendor of the software (sometimes empty).
product	String	The name of the software.
version	String	The version of the software.
patch	String	The patch level of the software, if any.
num_assets	Integer	The number of assets that have this software.
ip	String	The IP address where the software was found.
url	String	The URL where the software was found.
first_seen	Datetime	The first time we saw this software.
last_seen	Datetime	The last time we saw this software.

The Suggestions index

Sometimes we cannot accurately determine if a URL or IP is yours. If that happens it will appear in suggestions. You should regularly check this index and accept or reject the suggestions.

Field	Type	Description
url	String	The URL that is suggested.
ip	String	The IP address that is suggested.
related_to	String	The asset the suggestion is related to.
found	Datetime	The date and time when we found the suggestion.
rejected	Datetime	The date and time when you rejected the suggestion.
accepted	Datetime	The date and time when you accepted the suggestion.
first_seen	Datetime	The first time we saw this suggestion.
last_seen	Datetime	The last time we saw this suggestion.

The Suppliers index

This index contains a list of suppliers we found for your assets. It is useful when monitoring for supply chain attacks.

Field	Type	Description
name	String	The name of the supplier.
type	String	The type of the supplier (software, ISP, Certificate issuer, Saas platform, etc).
first_seen	Datetime	The first time we saw this supplier.
last_seen	Datetime	The last time we saw this supplier.

The Urls index

This index contains all the URLs we found for your assets. Urls are the unit that count for your assets and have websites and certificates associated with them.

Field	Type	Description
url	String	The URL.
tags	List of Strings	The ShadowTrackr tags for the URL.
first_seen	Datetime	The first time we saw this URL.
last_seen	Datetime	The last time we saw this URL.

The Websites index

This index contains all the websites we found for your assets. These can be urls or just ip addresses. If a website was found on a non standards port, like 8081, it will also be in this index. Websites can run multiple ip addresses. We'll monitor all of them.

Field	Type	Description
url	String	The URL of the website.
ip	String	The IP address of the website.
http_status	Integer	The HTTP status code of the website (200, 302, etc) without TLS.
http_error	String	An error that occurred while loading the website over HTTP, like a timeout or a connection error.
https_status	Integer	The HTTP status code of the website (200, 302, etc) with TLS.
https_error	String	An error that occurred while loading the website over HTTPS, like a timeout or a connection error.
grade	String	The security grade of the website. Possible values are A+, A, A-, B+, B, C+, C, D+, D, F, T, U, X, M, S and I.
tags	List of Strings	The ShadowTrackr tags for the website (often inherited from the URL).
first_seen	Datetime	The first time we saw the website.
last_seen	Datetime	The last time we saw the website.
cloud	Boolean	True if the website is hosted at a cloud or CDN provider. False if not.
cloudprovider	String	The name of the cloud provider.
defaced	Boolean	Set the True if we think the website is defaced.
google_analytics_id	String	The Google Analytics ID found on the website.
meta_pixel_id	String	The Meta pixel ID found on the website.
http_basic_authentication	Boolean	True if the website requires basic authentication over HTTP. False if not.
https_basic_authentication	Boolean	True if the website requires basic authentication over HTTPS. False if not.
http_cookie_on_load	Boolean	True if the website set cookies on load without asking over HTTP. False if not.
https_cookie_on_load	Boolean	True if the website set cookies on load without asking over HTTPS. False if not.
external_css	List of Strings	A list of links to external CSS dependencies.
external_scripts	List of Strings	A list of links to external JavaScript dependencies.
http_headers	String	The raw HTTP headers of the website over HTTP.
https_headers	String	The raw HTTP headers of the website over HTTPS.
http_server	String	The HTTP server header used by the website over HTTP.
https_server	String	The HTTP server header used by the website over HTTPS.
http_title	String	The title of the website over HTTP.
https_title	String	The title of the website over HTTPS.
http_redirect_history	List of Objects	A list of redirects encountered when accessing the website over HTTP.
https_redirect_history	List of Objects	A list of redirects encountered when accessing the website over HTTPS.
http_login_form	Boolean	True if a login form is present on the website over HTTP. False if not.
https_login_form	Boolean	True if a login form is present on the website over HTTPS. False if not.
http_login_insecure	Boolean	True if the login form uses and insecure connection over HTTP. False if not.
https_login_insecure	Boolean	True if the login form uses and insecure connection over HTTP. False if not.
jarm	String	The JARM of the website.
mmh3	Integer	The MurmurHash3 of the website favicon.
security_txt_exists	Boolean	True if a security.txt file is present on the website. False if not.
security_txt_valid	Boolean	True if the security.txt file is valid. False if not.
security_txt_errors	String	Errors encountered while validating the security.txt file, if any.
software	List of Strings	A list of software found on the website.
vulnerable	Boolean	True if a known vulnerability is present on the website. False if not.
vulnerabilities	List of Objects	A list of the vulnerabilities found on the website.
http_scripts	List of Strings	A list of scripts found on the website over HTTP.
https_scripts	List of Strings	A list of scripts found on the website over HTTPS.
http_css	List of Strings	A list of CSS files found on the website over HTTP.
https_css	List of Strings	A list of CSS files found on the website over HTTPS.
certificate_san	List of Strings	The Subject Alternative Names (SAN) found in the TLS certificate of the website.
no_url	Boolean	True if the website is accessed by IP only.
https_ssdeep_change	Boolean	True if the ssdeep hash of the website changed over HTTPS.
http_ssdeep_change	Boolean	True if the ssdeep hash of the website changed over HTTP.
warning	Boolean	True if the website is flagged as a warning. False if not.
problem	Boolean	True if the website is flagged as a problem. False if not.
problems	List of Strings	A description of the problems found on the website.
warnings	List of Strings	A description of the warnings found on the website.

The Vulnerabilities index

This index contains all vulnerabilities we found for your assets. Each entry is a single vulnerability on a single asset.

Field	Type	Description
cve	String	The CVE number of the vulnerability.
cvss_score	Float	The CVSS score of the vulnerability.
cvss_severity	String	The CVSS severity of the vulnerability (Low, Medium, High, Critical).
software	String	The software that is vulnerable.
asset	String	The IP address or URL of the asset.
first_seen	Datetime	The first time we saw this vulnerability on this asset.
last_seen	Datetime	The last time we saw this vulnerability on this asset.
exploited	Boolean	True if the vulnerability is known to be exploited in the wild.
tags	List of Strings	The ShadowTrackr tags for the assets.
description	String	A description of the vulnerability.

The Hibp index

This index contains all Have I Been Pwned (HIBP) breaches we found for your email addresses.

Field	Type	Description
email	String	The email address that was found in a breach.
breach	String	The name of the breach.
description	String	A description of the breach.
first_seen	Datetime	The first time we saw this breach for this email address.
last_seen	Datetime	The last time we saw this breach for this email address.
date	String	The date of the breach.

The Subnets index

This index contains all the subnets we found for your assets.

Field	Type	Description
cidr	String	The subnet range in CIDR notation (e.g., 1.2.3.0/24).
asn	Integer	The Autonomous System Number (ASN) of the subnet.
asn_name	String	The name of the ASN.
asn_country	String	The country of the ASN.
asn_allocated	String	The date when the ASN was allocated.
asn_registry	String	The registry of the ASN.
prefix	String	The BGP prefix of the subnet.
prefix_allocated	String	The date when the prefix was allocated.
prefix_registry	String	The registry of the prefix.
first_seen	Datetime	The first time we saw this subnet.
last_seen	Datetime	The last time we saw this subnet.
tags	List of Strings	The ShadowTrackr tags for the subnet.

The Cves index

This index contains all CVEs we know about.

Field	Type	Description
cve	String	The CVE number.
cvss_score	Float	The CVSS score of the CVE.
cvss_severity	String	The CVSS severity of the CVE.
cve_published	Datetime	The date the CVE was published.
vendor	String	The vendor of the software product.
product	String	The product name for a product affected by the CVE.
version	String	The version of the software.
exploited	Boolean	True if the CVE is known to be exploited in the wild.
cisa	Boolean	True if the CVE is on the CISA Known Exploited Vulnerabilities list.
cve_description	String	A description of the CVE from Mitre.
nvd_description	String	A description of the CVE from the National Vulnerability Database.

The Shadowserver_device_id index

This index contains device identification data from Shadowserver. This index only appears if you have the import enabled in the Shadowserver integration. If is an import from your Shadowserver account. Mote information on the data at Shadowserver.

Field	Type	Description
ip	String	The IP address of the device (can be IPv4 or IPv6.
timestamp	Datetime	The timestamp when Shadowserver saw this device.
severity	String	Low, medium, high or critical.
protocol	String	The protocol for the Shadowserver finding.
port	Integer	The port for the Shadowserver finding.
hostname	String	The hostname Shadowserver found, if any.
hostname_source	String	The source for the hostname.
tags	String	The tags Shadowserver gave the finding
asn	Integer	The ASN for the Shadowserver finding.
geo	String	The two letter country code for the Shadowserver finding.
region	String	The geographical region for the Shadowserver finding.
naics	Integer	The North American Industry Classification System (NAICS) code for ip owner.
sector	String	The business sector/branch the ip owner belongs to.
device_vendor	String	The vendor of the device.
device_model	String	The model of the device.
device_type	String	The type of the device.
timestamp	Datetime	The timestamp when Shadowserver saw this device.
first_seen	Datetime	Convenience field mapping to timestamp.
last_seen	Datetime	Convenience field mapping to timestamp.