File System Crawler
diskover crawls your storage servers locally or over NFS/SMB and scrapes file/directory meta data into Elasticsearch.
Visualize Your Storage
Identify old and unused files and give better insights into data change, duplicate files and wasted disk space.
-- linuxserver.io community memberThis is the first tool I've found that can index 7m files/2m directories in under 20 min
diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases.
As the amount of file data generated by businesses continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.
Using diskover, users can identify old and unused files and give better insights into data change, file duplication and wasted space. diskover supports crawling local file-systems or over NFS/SMB. Amazon S3 inventory files are also supported.
|OVA Files for VMware||Runs on Amazon AWS/S3 Support||Works with Docker|
|Patreon sponsors get access to OVA's which get diskover up and running quickly and easily. The OVA files can be imported into VMware, etc to get you crawling all your storage servers in less than an hour.||diskover works on AWS using EC2 and Elasticsearch instances. Crawl bots can run locally and push file system meta data into your AWS ES cluster. S3 Inventory files can also be imported into Elasticsearch to view S3 bucket usage.||Run diskover and diskover-web containers anywhere. Docker Hub images and Dockerfiles can be found on diskover github.|
diskover worker bots crawling file system (gource videos)