Unified Cloud Storage Management System

Orchestrate multiple storage solutions with intelligent metadata extraction and semantic search

Deploy Now Learn More

Key Features

☁️

Multi-Cloud Storage

Seamlessly connect and manage AWS S3, local filesystem, SQLite, PostgreSQL and more through one interface

🔍

Semantic Search

Automatically extract metadata from files and enable powerful semantic search capabilities

🔄

Dynamic Resources

Monitor external links (like YouTube) for changes and update metadata automatically

🔒

Advanced Controls

Built-in versioning, collision detection, and deduplication for robust data management

Microservice Architecture

🧩

Modular Design

Swap internal components like databases and storage backends with minimal configuration

🐳

Docker Deployment

All-in-one Docker image for quick local or cloud deployment

🐇

RabbitMQ Communication

Reliable microservice communication through RabbitMQ message queues for resilience and scalability

⚖️

Zero-Downtime Scaling

Add new metadata extractors and services dynamically without disrupting the running system

🔌

Pluggable Storage

Support for multiple storage providers that can be mixed and matched

📊

Metadata Pipeline

Extensible extraction system that analyzes files and external resources

Deployment

1. System Requirements

Platform: Linux (Ubuntu recommended), macOS, or Windows with WSL
Memory: Minimum 4GB RAM (8GB recommended)
Dependencies: Docker, Python 3.8+
Storage: 15GB for core system + storage requirements

2. Clone the repository

git clone https://github.com/xlukacs/omnivaultx.git

3. Run the core system

Run the start script to setup the core system.

./start_image.sh

Note: Save the url displayed in the terminal, you will need it to connect from the CLI.

4. Install CLI tool

As the CLI tool is distributed via a DEB package, you will need to install it manually:

sudo dpkg -i omnivaultx_cli_[version]_amd64.deb

5. Connect from the CLI

Set the API url and authenticate:

om --setapi [url:port]

om --authenticate

Now follow the instructions to authenticate.

om --help

Optional: Access the web interface

Configure the message broker for microservice communication:

The web interface is not yet available for local deployment. For a web version please use app.omnivaultx.com.

CLI Usage

Initialize & Configure

Set the API url.

The system can be configured to use multiple systems, for example a local system and a remote system.

om --setapi [url:port]

Authenticate

Authenticate with the system.

Authentication is done via a token flow. Authentication is only necessary for online systems.

om --authenticate

Test connection

Test the connection to the system.

om --test

Help command

Get help about the CLI.

om --help

List available connections

List all available connections.

om -lc

Connect to a connection

Connect to a connection.

om -c [connection_id]

Create a new connection

Create a new connection.

om -newcon [fs|s3|sq|[g]] ?([s3-bucket-name] [s3-access-key] [s3-secret-key] [s3-region]) ?([pg-host] [pg-port] [pg-user] [pg-password] [pg-db])

Delete a connection

Delete a connection.

om -delcon [connection_id]

View current connection details

om -constatus

List tags for a resource

List all tags for a resource.

om -tag-ls [path]

Add tag to a resource

Add a tag to a resource. This tag can be used to filter resources.

om -tag-add [path] [tag]

Remove tag from a resource

Remove a tag from a resource. This tag wont be available for filtering.

om -tag-rm [path] [tag]

Edit tag for a resource

Rename a tag for a resource.

om -tag-edit [path] [tagOld] [tagNew]

List all resources

List all the resources in the current remote directory.

om -ls

Change directory

Change the current remote directory.

om -cd [path]

Copy a resource

Copy a resource from pathA to pathB on the remote system.

om -cp [pathA] [pathB]

Move a resource

Move a resource from pathA to pathB on the remote system.

om -mv [pathA] [pathB]

Delete a resource

Delete a resource from the remote system on a specific path.

om -del [path]

Upload a resource

Upload a resource to the remote system on a specific path.

The remote path will be the current remote directory.

om --upload [path]

Create a new directory

Create a new directory on the remote system on a specific path.

The remote path will be the current remote directory.

om -mkdir [path]

Change internal database

In local deployment, you can change the internal database engine from sqlite to postgres.

om -set-meta-engine [sqlite|postgres] ?(connectionString_for_pg)

Synchronization

Turn on local operations for resources.

om -sync-on

Turn off local operations for resources.

om -sync-off

View sync status

View the sync status of the resources.

om -sync-status

CLI mode

Override system commands with CLI commands.

om --activate

om --deactivate

Screenshots

Connection dashboard

The connection dashboard shows the current user, the already defined connections and an option to add a new one.

Resource input and search

The resource input and search feature allows you to search for resources by name or tag. Support boolean search.

File explorer

The file explorer shows the current directory and the resources in it. You can navigate through the directory structure and select a resource to view its details. Tag management is also available here.

Dynamic resource scanner

The dynamic resource scanner allows you to enter an URL and do a dynamic scan. The returned resource type will be automatically selected, you just need to name the resource.

CLI installation

The CLI can be installed via dpkg. It will be installed in the /etc/om directory, optionally you can add a provided shell script to your PATH to see what is the connected remote directory.

CLI command examples

The CLI supports many commands. Here is an example of the list command.

Core system local startup

These are the logs of the core system, while starting up. The system will log all the IP addresses it is listening on.

CLI authentication

The CLI supports authentication via token flow. Enter the provided url and after typing the token enter your credentials.

Contact Me

Get in Touch

Interested in implementing my research or discussing potential applications? Feel free to reach out.

📧

Email

[email protected]

🔗

linkedin.com/

💻

GitHub

github.com/xlukacs