Integrate Storage
Introduction
Storage plays an essential role in machine learning deployment, without storage, the machine learning algorithm won't be able to learn and produce an accurate model.
Cortex supports wide variety of storage technologies and file formats. Our platform architecture is modular and flexible to accommodate your custom, proprietary storage technology.
There is no limitation on how large Cortex can ingest your data. The limitation is bound by hardware specification.
Database
Database is an important storage technology in enterprise. Its distributed processing, high availability and scalability have been proven to be an indispensable data store in the enterprise environment.
We support both major database technologies:
- relational database
- non-relational database
Relational Database
If you are storing data in well structured way (schema based) and persist the data for long term objective, Cortex will ingest your data stored in relational database out of the box. As Cortex is based on Java architecture, basically we support almost all relational databases that is in used in the industry today.
In the event your database is proprietary and not supported out of the box, we are able to create custom connector for you.
Below are some databases we support:
- MySQL
- Microsoft SQL Server
- Oracle
- PostgreSQL
Non-Relational Database
Sometimes, you have data stored in unstructured way or more flexible or non-schema based database with non-relational database. You also persist the data for long term objectives.
Though relational database is still more widely used in the industry, however non-relational database is gaining momentum.
Cortex support data ingestion from non-relational database out of the box.
Below are some non-relational database we support:
- Cassandra
- MongoDB
File
You may have data store in a particular file format. We support popular file format targeted for different types of usage. Some of these files may have been produced by another machines/Internet of Thing device and may be high in volume and varieties.
Cortex support file ingestion out of the box. If you have a file encoded in proprietary format, talk to us, we will create a custom file reader for you.
Below are some of the file formats we support:
Comma Separated Value (CSV)
Javascript Object Notation (JSON)
XML (Extensible Markup Language)
Image files (PNG, JPEG)
Video files (MPEG)
Live Data Stream
Cortex support live data streams that offer as an extension of Cortex Core package. It offers scalable, high-throughput, fault-tolerant stream processing in live data streams. Data can be ingested from many sources such as message queue, IoT devices, transactional processing system (fraud detection system, aviation monitoring, emergency dispatch system).
Cortex will processed these live data using complex algorithm. And processed data can be pushed out to filesystems, databases and live dashboards.
Create Storage
First, we need to connect to your storage system.
- On the left navigational bar, click Storage
- An expanded, drop-down menu will be shown
- Click My Storage
My Storage screen will be displayed subsequently.
To create a storage, click on Create CassandraDB button. A dialog box will appear
- You may assigned any name to be Storage name to represent this storage device in Cortex.
- Put in a short description to explain the purpose of this storage device
- Key in host name or IP address for the storage device to be connected to
- Key in port number on where the target storage device has the storage service running
- Key in username and password to authenticate the remote storage device. Beware username and password are case sensitive.
- Click Connect button to verify connection to the storage device is established
- Click Create to create storage in Cortex
A storage will be created.