Unified Cloud Storage Management System

Orchestrate multiple storage solutions with intelligent metadata extraction and semantic search

Key Features

☁️

Multi-Cloud Storage

Seamlessly connect and manage AWS S3, local filesystem, SQLite, PostgreSQL and more through one interface

🔍

Semantic Search

Automatically extract metadata from files and enable powerful semantic search capabilities, including tag-based filtering

🔄

Dynamic Resources

Monitor external links (like YouTube) for changes and update metadata automatically. Rescan files on demand through web or CLI

🔒

Advanced Controls

Built-in versioning, collision detection, and deduplication for robust data management

🚀

Extendible system

Bring your own META extractors, based on instructions here

Microservice Architecture

🧩

Modular Design

Swap internal components like databases, storage backends and metadata extractors with minimal configuration

🐳

Docker Deployment

All-in-one Docker image for quick local or cloud deployment

🐇

RabbitMQ Communication

Reliable microservice communication through RabbitMQ message queues for resilience and scalability

⚖️

Zero-Downtime Scaling

Add new metadata extractors and services dynamically without disrupting the running system

🔌

Pluggable Storage

Support for multiple storage providers that can be mixed and matched

📊

Metadata Pipeline

Extensible extraction system that analyzes files and external resources. Connect your own custom meta extractors through RabbitMQ

Deployment

1. System Requirements

Platform: Linux (Ubuntu recommended), macOS, or Windows with WSL
Memory: Minimum 4GB RAM (8GB recommended)
Dependencies: Docker, Python 3.8+
Storage: 15GB for core system + storage requirements

2. Clone the repository

git clone https://github.com/xlukacs/omnivaultx.git

3. Run the core system

Run the start script to setup the core system.

./start_image.sh

Note: Save the url displayed in the terminal, you will need it to connect from the CLI.

4. Install CLI tool

As the CLI tool is distributed via a DEB package, you will need to install it manually:

sudo dpkg -i om_[version]_amd64.deb

5. Connect from the CLI

Set the API url and authenticate:

om --setapi [url:port]

om --authenticate

Now follow the instructions to authenticate. Use the -noauth flag to disable auth checks in the cli.

6. Connect meta extractors

Download the meta extractors from the /extractors folder on github and connect them:

python -m venv venv

source venv/bin/activate

pip install -r requirements.txt

Rename the .env.exmaple file to .env and set the variables. When done, run the script:

python main.py

Note: The default username and password are dp-processor and dp-processor.

Optional: Access the web interface

Configure the message broker for microservice communication:

The web interface is not yet available for local deployment. For a web version please use app.omnivaultx.com.

CLI Usage

Initialize & Configure

Set the API url.

The system can be configured to use multiple systems, for example a local system and a remote system.

om --setapi [url:port]

Authenticate

Authenticate with the system.

Authentication is done via a token flow. Authentication is only necessary for online systems.

om --authenticate

When connecting to a local system where auth is disabled, use the -noauth flag to disable local auth checks.

Test connection

Test the connection to the system.

om --test

Help command

Get help about the CLI.

om --help

List available connections

List all available connections.

om -lc

Connect to a connection

Connect to a connection.

om -con [connection_id]

Create a new connection

Create a new connection.

om -newcon [fs|s3|sq|[g]] ?([s3-bucket-name] [s3-access-key] [s3-secret-key] [s3-region]) ?([pg-host] [pg-port] [pg-user] [pg-password] [pg-db])

Delete a connection

Delete a connection.

om -delcon [connection_id]

View current connection details

om -constatus

List tags for a resource

List all tags for a resource.

om -tag-ls [path]

Add tag to a resource

Add a tag to a resource. This tag can be used to filter resources.

om -tag-add [path] [tag]

Remove tag from a resource

Remove a tag from a resource. This tag wont be available for filtering.

om -tag-rm [path] [tag]

Edit tag for a resource

Rename a tag for a resource.

om -tag-edit [path] [tagOld] [tagNew]

Rescan a resource

Resubmit a resource for processing.

om --rescan [path]

List all resources

List all the resources in the current remote directory.

om -ls

Change directory

Change the current remote directory.

om -cd [path]

Display current directory

Display the current remote directory.

om -pwd

Copy a resource

Copy a resource from pathA to pathB on the remote system.

om -cp [pathA] [pathB]

Move a resource

Move a resource from pathA to pathB on the remote system.

om -mv [pathA] [pathB]

Delete a resource

Delete a resource from the remote system on a specific path.

om -del [path]

Upload a resource

Upload a resource to the remote system on a specific path.

The remote path will be the current remote directory.

om --upload [path]

Enter a dynamic resource to the system with the following command:

om --upload -link [url] -name [resource_name]

Download a resource

Download a resource from the remote system to the current local directory.

The remote path will be the current remote directory.

om --download [path]

Search for a resource

Search for a resource by name or tag.

om --search [name|tag]

Note: Boolean operators are supported (AND, OR, NOT).

Create a new directory

Create a new directory on the remote system on a specific path.

The remote path will be the current remote directory.

om -mkdir [path]

Change internal database

In local deployment, you can change the internal database engine from sqlite to postgres.

om -set-meta-engine [sqlite|postgres] ?(connectionString_for_pg)

Synchronization

Turn on local operations for resources.

om -sync-on

Turn off local operations for resources.

om -sync-off

View sync status

View the sync status of the resources.

om -sync-status

CLI mode

Override system commands with CLI commands.

om --activate

om --deactivate

Auth status

Turn on/off local auth checks.

om -noauth

om -auth

Screenshots

Connection dashboard

The connection dashboard shows the current user, the already defined connections and an option to add a new one.

Resource input and search

The resource input and search feature allows you to search for resources by name or tag. Support boolean search.

File explorer

The file explorer shows the current directory and the resources in it. You can navigate through the directory structure and select a resource to view its details. Tag management is also available here.

Dynamic resource scanner

The dynamic resource scanner allows you to enter an URL and do a dynamic scan. The returned resource type will be automatically selected, you just need to name the resource.

CLI installation

The CLI can be installed via dpkg. It will be installed in the /etc/om directory, optionally you can add a provided shell script to your PATH to see what is the connected remote directory.

CLI command examples

The CLI supports many commands. Here is an example of the list command.

Core system local startup

These are the logs of the core system, while starting up. The system will log all the IP addresses it is listening on.

CLI authentication

The CLI supports authentication via token flow. Enter the provided url and after typing the token enter your credentials.

Contact Me

Get in Touch

Interested in implementing my research or discussing potential applications? Feel free to reach out.

📧

Email

[email protected]

🔗

linkedin.com/

💻

GitHub

github.com/xlukacs

Unified Cloud Storage Management System

Key Features

Multi-Cloud Storage

Semantic Search

Dynamic Resources

Advanced Controls

Extendible system

Microservice Architecture

Modular Design

Docker Deployment

RabbitMQ Communication

Zero-Downtime Scaling

Pluggable Storage

Metadata Pipeline

Deployment

1. System Requirements

2. Clone the repository

3. Run the core system

4. Install CLI tool

5. Connect from the CLI

6. Connect meta extractors

Optional: Access the web interface

CLI Usage

Initialize & Configure

Authenticate

Test connection

Help command

List available connections

Connect to a connection

Create a new connection

Delete a connection

View current connection details

List tags for a resource

Add tag to a resource

Remove tag from a resource

Edit tag for a resource

Rescan a resource

List all resources

Change directory

Display current directory

Copy a resource

Move a resource

Delete a resource

Upload a resource

Download a resource

Search for a resource

Create a new directory

Change internal database

Synchronization

View sync status

CLI mode

Auth status

Screenshots

Connection dashboard

Resource input and search

File explorer

Dynamic resource scanner

CLI installation

CLI command examples

Core system local startup

CLI authentication

Contact Me

Get in Touch

Email

LinkedIn

GitHub