Data Collector Blueprints

The Open Data Hub leverages a microservices-based architecture to make data ingestion flexible and scalable. Data Collector Blueprints are pre-built, reusable microservices designed to simplify the process of bringing data into the platform. These blueprints offer a "low-code" approach, allowing you to configure existing collectors for your specific needs without writing extensive code or managing complex deployments.

📄️ Rest Push Collector

The Rest Push collector acts as a generic endpoint where external data providers can push their data directly to the Open Data Hub. This means you, as a data provider, don't need to deploy any additional software or agents on your side. Instead, you'll be provided with specific credentials and a URL, and you can simply send your data via standard HTTP POST requests.

📄️ MQTT Client Collector

A pre-built microservice designed to easily integrate with data sources that publish information via the MQTT protocol.

📄️ S3 Poller Collector

Designed for efficiently collecting data from AWS S3 buckets. This collector is ideal for scenarios where your data provider regularly uploads files to an S3 bucket, and you need to automatically retrieve these files and feed their content into the Open Data Hub pipeline.

📄️ API Crawler Collector

The API Crawler collector is designed to pull data from external APIs based on a declarative configuration. It allows for complex and dynamic API interactions, including multi-step calls, data transformations, and context-based processing. This collector is ideal for scenarios where data needs to be fetched from various API endpoints, potentially requiring pagination, authentication, or nested requests.