Skip to content

Latest commit

 

History

History
executable file
·
435 lines (402 loc) · 43.1 KB

Osint.md

File metadata and controls

executable file
·
435 lines (402 loc) · 43.1 KB

Open Source Intelligence

Table of Contents

Sort

https://github.com/woj-ciech/kamerka https://github.com/SourcingDenis/free-online-competitive-intelligence/blob/master/README.md https://github.com/0days/Blue https://github.com/digininja/leakyrepo

https://github.com/GeneralTesler/deluxe

  • Weaponizing Corporate Intel - Mike Felch and Beau Bullock(B-Sides Orlando 2019)

    • Strategically targeting a corporation requires deep knowledge of their technologies and employees. Successfully compromising an organization can depend on the quality of reconnaissance a tester performs up front. Often times testers only resort to using publicly available tools which can overlook critical assets. In this presentation, we will begin by examining some commonly overlooked methods to discover external resources. Next, we will show how to discover employees of a target organization and quickly locate their social media accounts. Finally, we will strategically identify and weaponize personal information about the employees to target the organization directly using new attack techniques. Attendees will learn an external defense evasion method, a new process to gain credentialed access, and be the first to receive a newly released tool! While the approach is designed to assist offensive security professionals, the presentation will be informative for technical and non-technical audiences; demonstrating the importance of security-awareness for everyone.

  • ODIN

Remove hidden data and personal information by inspecting documents, presentations, or workbooks https://support.office.com/en-us/article/remove-hidden-data-and-personal-information-by-inspecting-documents-presentations-or-workbooks-356b7b5d-77af-44fe-a07f-9aa4d085966f

  • yar
    • yar is an OSINT tool for reconnaissance of repositories/users/organizations on Github. Yar clones repositories of users/organizations given to it and goes through the whole commit history in order of commit time, in search for secrets/tokens/passwords, essentially anything that shouldn't be there. Whenever yar finds a secret, it will print it out for you to further assess. Yar searches either by regex, entropy or both, the choice is yours. You can think of yar as a bigger and better truffleHog, it does everything that truffleHog does and more!

General

  • General
    • SWOT - Strengths, Weaknesses, Opportunities, Threats
  • 101
  • Articles/Writeups
  • Alerting
    • Google Trends
      • See what are the popular related topics people are searching for. This will help widen your search scope.
    • Google Alerts
      • Google Alerts are email updates of the latest relevant Google results (web, news, etc.) based on your queries.
    • PasteLert
      • PasteLert is a simple system to search pastebin.com and set up alerts (like google alerts) for pastebin.com entries. This means you will automatically recieve email whenever your term(s) is/are found in new pastebin entries!
  • Educational
  • OSINT Based News
    • JustSecurity
      • Just Security is an online forum for the rigorous analysis of U.S. national security law and policy. We aim to promote principled and pragmatic solutions to national security problems that decision-makers face. Our Board of Editors includes individuals with significant government experience, civil society attorneys, academics, and other leading voices. Just Security is based at the Center for Human Rights and Global Justice at New York University School of Law.
    • OSINTInsight
    • Janes
    • bell?ngcat
      • By and for citizen investigative journalists
    • NightWatch
      • NightWatch is an executive commentary and analysis of events that pose or advance threats to US national security interests. It is deliberately edgy in the interest of clarity and brevity. As a product for executives, the distribution and all feedback comments are anonymous.
    • RSOE EDIS - Emergency and Disaster Information Service
  • Resources
  • Writeups
  • Talks & Presentations
  • OSINT Tools/Resources
    • Tools
      • DNS
      • All-in-One
        • Maltego
          • Description: What you use to tie everything together.
        • Oryon C Portable
          • Oryon C Portable is a web browser designed to assist researchers in conducting Open Source Intelligence investigations. Oryon comes with dozens of pre-installed tools and a select set of links cataloged by category – including those that can be found in the OI Shared Resources.
        • OSINT Mantra
        • Recon-ng
          • Description: Recon-ng is a full-featured Web Reconnaissance framework written in Python. Complete with independent modules, database interaction, built in convenience functions, interactive help, and command completion, Recon-ng provides a powerful environment in which open source web-based reconnaissance can be conducted quickly and thoroughly.
        • TouchGraph SEO Browser
          • Use this free Java application to explore the connections between related websites.
        • Th3inspector
          • Tool that automates OSINT collection. Seems to gather from a variety of sources. Perl script.
        • gasmask
          • All in one Information gathering tool - OSINT
      • Certificate Transparency
        • ct-exposer
          • An OSINT tool that discovers sub-domains by searching Certificate Transparency logs. Certificate Transparency (CT) is an experimental IETF standard. The goal of it was to allow the public to audit which certificates were created by Certificate Authorities (CA). TLS has a weakness that comes from the large list of CAs that your browser implicitly trusts. If any of those CAs were to maliciously create a new certificate for a domain, your browser would trust it. CT adds benefits to TLS certificate trust: Companies can monitor who is creating certificates for the domains they own. It also allows browsers to verify that the certificate for a given domain is in the public log record. These logs end up being a gold mine of information for penetration testers and red teams.
      • Data Manipulation
        • Danger-zone
          • Correlate data between domains, ips and email addresses, present it as a graph and store everything into Elasticsearch and JSON files.
          • Article
        • OpenRefine
          • Description: OpenRefine is a power tool that allows you to load data, understand it, clean it up, reconcile it to master database, and augment it with data coming from Freebase or other web sources. All with the comfort and privacy of your own computer.
        • OSRFramework
          • OSRFramework is a GNU AGPLv3+ set of libraries developed by i3visio to perform Open Source Intelligence tasks. They include references to a bunch of different applications related to username checking, DNS lookups, information leaks research, deep web search, regular expressions extraction and many others. At the same time, by means of ad-hoc Maltego transforms, OSRFramework provides a way of making these queries graphically as well as several interfaces to interact with like OSRFConsole or a Web interface.
      • Geolocation
        • Creepy.py
          • Description: Creepy is a geolocation OSINT tool. Gathers geolocation related information from online sources, and allows for presentation on map, search filtering based on exact location and/or date, export in csv format or kml for further analysis in Google Maps.
      • Research Collection/Organization
      • Search Engine
        • shodan.io
          • Description: DescriptionShodan is a search engine that lets the user find specific types of computers connected to the internet using a variety of filters.
  • Company/People Searching
    • data.com
    • LittleSis
      • LittleSis is a free database of who-knows-who at the heights of business and government.
    • Jigsaw
      • Jigsaw is a prospecting tool used by sales professionals, marketers and recruiters to get fresh and accurate sales leads and business contact information.
    • Spokeo
      • Spokeo is a people search engine that organizes white pages listings, public records and social network information into simple profiles to help you safely find and learn about people.\
    • Hoovers
      • Search over 85 million companies within 900 industry segments; Hoover's Reports Easy-to-read reports on key competitors, financials, and executives
    • Market Visual
      • Search Professionals by Name, Company or Title
    • Glass Door
      • Search jobs then look inside. Company salaries, reviews, interview questions, and more all posted anonymously by employees and job seekers.
    • 192
      • Find people, businesses and places in the UK with 192.com. Directory enquiries, a people finder, business listings and detailed maps with aerial photos.
    • corporationwiki
    • orbis
      • Company information across the globe
  • Country Specific Resources
  • CVS/Git/Similar Focused
    • repo-supervisor
    • GitPrey
      • GitPrey is a tool for searching sensitive information or data according to company name or key word something.The design mind is from searching sensitive data leakling in Github:
    • git-all-secrets
      • A tool to capture all the git secrets by leveraging multiple open source git searching tools
    • github-firehose
    • Gitem
      • Gitem is a tool for performing Github organizational reconnaissance.
    • Truffle Hog
      • Searches through git repositories for high entropy strings, digging deep into commit history and branches. This is effective at finding secrets accidentally committed that contain high entropy.
    • dvcs-ripper
      • Rip web accessible (distributed) version control systems: SVN, GIT, Mercurial/hg, bzr, ... It can rip repositories even when directory browsing is turned off.
    • Truffle Hog
      • Searches through git repositories for high entropy strings, digging deep into commit history
    • DVCS-Pillage
      • Pillage web accessible GIT, HG and BZR repositories. I thought it would be useful to automate some other techniques I found to extract code, configs and other information from a git,hg, and bzr repo's identified in a web root that was not 100% cloneable. Each script extracts as much knowledge about the repo as possible through predictable file names and known object hashes, etc.
    • gitdigger
      • gitDigger: Creating realworld wordlists from github hosted data.
    • gitrob
      • Gitrob is a command line tool which can help organizations and security professionals find sensitive information lingering in publicly available files on GitHub. The tool will iterate over all public organization and member repositories and match filenames against a range of patterns for files that typically contain sensitive or dangerous information. Looking for sensitive information in GitHub repositories is not a new thing, it has been known for a while that things such as private keys and credentials can be found with GitHub's search functionality, however Gitrob makes it easier to focus the effort on a specific organization.
    • reposcanner
      • Python script to scan Git repos for interesting strings
    • gitleaks
      • Searches full repo history for secrets and keys
    • Reposcanner
      • Reposcanner is a python script to search through the commit history of Git repositories looking for interesting strings such as API keys, inspired by truffleHog.
  • DNS Stuff
  • Domain Recon
    • Tools
      • Waybackpack
        • Waybackpack is a command-line tool that lets you download the entire Wayback Machine archive for a given URL.
      • domain - jhaddix
        • Recon-ng and Alt-DNS are awesome. This script combines the power of these tools with the ability to run multiple domains within the same session. TLDR; I just want to do my subdomain discovery via ONE command and be done with it. Only 1 module needs an api key (/api/google_site) find instructions for that on the recon-ng wiki. Script to enumerate subdomains, leveraging recon-ng. Uses google scraping, bing scraping, baidu scraping, yahoo scraping, netcraft, and bruteforces to find subdomains. Plus resolves to IP
      • check0365
        • checkO365 is a tool to check if a target domain is using O365
  • Email Gathering/Reconnaissance
    • Articles/Writeups
    • Tools
      • SimplyEmail
        • What is the simple email recon tool? This tool was based off the work of theHarvester and kind of a port of the functionality. This was just an expansion of what was used to build theHarvester and will incorporate his work but allow users to easily build Modules for the Framework. Which I felt was desperately needed after building my first module for theHarvester.
      • Email Reconnaissance and Phishing Template Generation Made Simple
      • theHarvester
        • theHarvester is a tool for gathering e-mail accounts, subdomain names, virtual hosts, open ports/ banners, and employee names from different public sources (search engines, pgp key servers).
      • discover.sh
        • For use with Kali Linux. Custom bash scripts used to automate various pentesting tasks.
      • Cr3dOv3r
        • Cr3dOv3r simply you give it an email then it does two simple jobs (but useful): Search for public leaks for the email and if it any, it returns with all available details about the leak (Using hacked-emails site API). Now you give it this email's old or leaked password then it checks this credentials against 16 websites (ex: facebook, twitter, google...) then it tells you if login successful in any website!
      • Infoga
        • Infoga is a tool gathering email accounts informations (ip,hostname,country,...) from different public source (search engines, pgp key servers and shodan) and check if emails was leaked using haveibeenpwned.com API. Is a really simple tool, but very effective for the early stages of a penetration test or just to know the visibility of your company in the Internet.
  • Facial Mapping Data
    • Social Mapper
      • Social Mapper is a Open Source Intelligence Tool that uses facial recognition to correlate social media profiles across different sites on a large scale. It takes an automated approach to searching popular social media sites for targets names and pictures to accurately detect and group a person’s presence, outputting the results into report that a human operator can quickly review.
  • Fancy Search Engines
    • Entity Cube
      • EntityCube is a research prototype for exploring object-level search technologies, which automatically summarizes the Web for entities (such as people, locations and organizations) with a modest web presence.
    • Silobreaker
      • Enterprise Semantic Search Engine, allows virtualisation of data, analytics and exploration of key data.
    • iSeek
      • Another handy search engine that break results down into easy to manage categories.
    • Carrot2
      • Carrot2 organizes your search results into topics. With an instant overview of what's available, you will quickly find what you're looking for.
    • Sqoop
      • OSINT search engine of public documents(handy)
    • GlobalFileSearch
      • An FTP Search Engine that may come in handy.
    • NAPALM FTP Indexer
  • General Meta Data
    • Just-Metadata
      • Just-Metadata is a tool that can be used to gather intelligence information passively about a large number of IP addresses, and attempt to extrapolate relationships that might not otherwise be seen. Just-Metadata has "gather" modules which are used to gather metadata about IPs loaded into the framework across multiple resources on the internet. Just-Metadata also has "analysis" modules. These are used to analyze the data loaded Just-Metadata and perform various operations that can identify potential relationships between the loaded systems.
    • MetaGooFil
      • Description: Metagoofil is an information gathering tool designed for extracting metadata of public documents (pdf,doc,xls,ppt,docx,pptx,xlsx) belonging to a target company. The tool will perform a search in Google to identify and download the documents to local disk and then will extract the metadata with different libraries like Hachoir, PdfMiner and others. With the results it will generate a report with usernames, software versions and servers or machine names that will help Penetration testers in the information gathering phase.
    • Metashield Analyzer
      • Description: Metadata documents can help a malicious user to obtain information that is beyond our control in an enterprise environment. Metashield Analyzer is an online service that allows easily check if your office documents contain metadata.
    • PowerMeta
      • PowerMeta searches for publicly available files hosted on various websites for a particular domain by using specially crafted Google, and Bing searches. It then allows for the download of those files from the target domain. After retrieving the files, the metadata associated with them can be analyzed by PowerMeta. Some interesting things commonly found in metadata are usernames, domains, software titles, and computer names.
  • General Data Scrapers
    • XRAY
      • XRay is a tool for recon, mapping and OSINT gathering from public networks.
    • NameCheck
      • Search usernames across multiple services/domain registries
    • [TheHarvester](From: https://code.google.com/p/theharvester/)
      • Description: The objective of this program is to gather emails, subdomains, hosts, employee names, open ports and banners from different public sources like search engines, PGP key servers and SHODAN computer database. This tool is intended to help Penetration testers in the early stages of the penetration test in order to understand the customer footprint on the Internet. It is also useful for anyone that wants to know what an attacker can see about their organization.
    • OSINT OPSEC Tool
      • Description: The OSINT OPSEC Tool monitors multiple 21st Century OSINT sources real-time for keywords, then analyses the results, generates alerts, and maps trends of the data, finding all sorts of info people probably don't want others to see...
    • Pattern
      • Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google,; Twitter, Wikipedia), web crawler, HTML DOM parser; Natural Language Processing: part-of-speech taggers, n-gram search, sentiment analysis, WordNet; Machine Learning: vector space model, clustering, classification (KNN, SVM, Perceptron); Network Analysis: graph centrality and visualization.
    • Paste-Site Scrapers
      • sniff-paste
        • Multithreaded pastebin scraper, scrapes to mysql database, then reads pastes for noteworthy information.
  • Search Engine Dorks
  • Network Information Search Engines
    • Whoisology
      • Whoisology is a domain name ownership archive with literally billions of searchable and cross referenced domain name whois records.
  • Site Specific
    • AWS
      • AWSBucketDump
        • AWSBucketDump is a tool to quickly enumerate AWS S3 buckets to look for loot. It's similar to a subdomain bruteforcer but is made specifically for S3 buckets and also has some extra features that allow you to grep for delicious files as well as download interesting files if you're not afraid to quickly fill up your hard drive.
    • Facebook
      • pymk-inspector
        • The pymk-inspector is a tool built by Gizmodo's Special Projects Desk that we used for our investigation into Facebook's people you may know (pymk) algorithm.
      • Find FB profiles by Email
    • Github
    • LinkedIn
      • InSpy
        • A LinkedIn enumeration tool
      • linkedin
        • Linkedin Scraper using Selenium Web Driver, Firefox 45, Ubuntu and Scrapy
      • LinkedInt: A LinkedIn scraper for reconnaissance during adversary simulation
      • LinkedIn Gatherer
      • socilab
        • This site allows users to visualize and analyze their LinkedIn network using methods derived from social-scientific research. Full sample output is shown here. The site is free and open-source. Have fun!
      • Linkedin_profiles
        • This script uses selenium to scrape linkedin employee details from a specified company. If the script isn't working, you can always browse to the desired company's employee page and paste in the link on line 69 like this: "employees_page = url"
      • The Secrets of LinkedIn
        • Grabbing usernames/connections(link analysis)
      • The Endorser
        • An OSINT tool that allows you to draw out relationships between people on LinkedIn via endorsements/skills.
      • ScrapedIn
        • this tool assists in performing reconnaissance using the LinkedIn.com website/API. Provide a search string just as you would on the original website and let ScrapedIn do all the dirty work. Output is stored as an XLSX file, however it is intended to be used with Google Spreadsheets. After importing the XLSX into Google Spreadsheets there will be a "dataset" worksheet and a "report" worksheet.
      • Gathering Usernames from Google LinkedIn Results Using Burp Suite Pro - BHIS
      • GatherContacts
        • A Burp Suite Extension to pull Employee Names from Google and Bing LinkedIn Search Results.
      • linkedin2username
      • Raven
        • raven is a Linkedin information gathering tool that can be used by pentesters to gather information about an organization employees using Linkedin.
    • Tinder
    • Twitter
      • OneMillionTweetMap
        • This page maps the last geolocalized tweets delivered by the twitter stream API. ... YES - IN REAL-TIME - and we keep "only" the last one million tweets.
      • tweets_analyzer
        • Tweets metadata scraper & activity analyzer
      • Tweet Archivist
      • tweets_analyzer
        • Tweets metadata scraper & activity analyzer
      • Tinfoleak
        • tinfoleak is a simple Python script that allow to obtain: basic information about a Twitter user (name, picture, location, followers, etc.); devices and operating systems used by the Twitter user; applications and social networks used by the Twitter user; place and geolocation coordinates to generate a tracking map of locations visited; show user tweets in Google Earth!; download all pics from a Twitter user; hashtags used by the Twitter user and when are used (date and time); user mentions by the the Twitter user and when are occurred (date and time); topics used by the Twitter user
      • How to Find the Twitter ID from an Email Address - booleanstrings.com
      • Twint
        • Formerly known as Tweep, Twint is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API. Twint utilizes Twitter's search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends, or sort out sensitive information from Tweets like e-mail and phone numbers. I find this very useful, and you can get really creative with it too. Twint also makes special queries to Twitter allowing you to also scrape a Twitter user's followers, Tweets a user has liked, and who they follow without any authentication, API, Selenium, or browser emulation.
      • twitterBFTD
        • Twitter back from the death looks in a user tweets history for domain names that are available for registration.
        • Blogpost
  • Social Media Search/Enumeration
    • CheckUsernames
      • Check the use of your brand or username on 160 Social Networks
    • NameCHK
      • Check to see if your desired username or vanity url is still available at dozens of popular Social Networking and Social Bookmarking websites.
    • Scythe
      • The ability to test a range of email addresses across a range of sites (e.g. social media, blogging platforms, etc...) to find where those targets have active accounts. This can be useful in a social engineering test where you have email accounts for a company and want to list where these users have used their work email for 3rd party web based services.
    • Social Mention
      • Social Mention is a social media search engine that searches user-generated content such as blogs, comments, bookmarks, events, news, videos, and more
    • Whos Talkin
      • social media search tool that allows users to search for conversations surrounding the topics that they care about most.
    • sherlock-js
      • Find usernames across over 75 social networks - NodeJS remake of sdushantha/sherlock
    • sherlock
      • Python tool to find usernames across social networks
  • Tor
    • ExoneraTor
      • Enter an IP address and date to find out whether that address was used as a Tor relay: