Support the development  Sponsor Patreon  Donate PayPal

file system crawler

File System Crawler

diskover crawls your storage servers locally or over NFS/SMB and scrapes file/directory meta data into Elasticsearch.

Learn More

analyze metadata

Visualize Your Storage

Identify old and unused files and give better insights into data change, duplicate files and wasted disk space.

Learn More

open-source

Open Source

Built using Python + PHP + HTML5 + Javascript + D3.js. diskover runs on Linux and macOS.

Learn More

This is the first tool I've found that can index 7m files/2m directories in under 20 min

-- linuxserver.io community member

diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases.

As the amount of file data generated by business' continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.

Using diskover, users can identify old and unused files and give better insights into data change, file duplication and wasted space. diskover supports crawling local file-systems or over NFS/SMB. Amazon S3 inventory files are also supported.

diskover diagram

Screenshots

See data change on your file system and identify hot spots
diskover-web hotdirs
Visualize your file system using one of the many analytics
diskover-web filetree
Tag files and directories using default and custom tags, export file lists
diskover-web tagging
Use the built-in rest-api to assist with tagged data cleanup/moving
diskover-web tags
Use pre-made or create custom smart searches
diskover-web tagging
Find duplicate files taking up disk space
diskover-web dupes
Quickly search all your storage servers
diskover-web file search
Use Elasticsearch query syntax to find files and directories
diskover-web file search es query
OVA Files for VMware Runs on Amazon AWS/S3 Support Works with Docker
diskover ova vmware diskover aws diskover docker
Patreon sponsors get access to OVA's which get diskover up and running quickly and easily. The OVA files can be imported into VMware, etc to get you crawling all your storage servers in less than an hour. diskover works on AWS using EC2 and Elasticsearch instances. Crawl bots can run locally and push file system meta data into your AWS ES cluster. S3 Inventory files can also be imported into Elasticsearch to view S3 bucket usage. Run diskover and diskover-web containers anywhere. Docker Hub images and Dockerfiles can be found on diskover github.

diskover worker bots crawling file system (gource videos)