Big Data Analysis without the Cloud Complexity
The future of Earth Data Science analysis is in the Cloud and with Earth Big Data you are in control from development to operations.
Managing Cloud Resources
Recipe
Management
Assign compute and storage resources to each step of a recipe
Prioritize recipe steps
Run recipe steps in parallel
Have recipe steps execute only after other steps have completed
Credentials
Management
Propagate user credentials in a secure and efficient method
Mimic the familiar Linux dotfile approach
Assign identities such as administrator, super user, and captain
Workflow
Monitoring
Monitor all aspects on how recipes are progressing
See which recipe steps and resources are active
Handle exceptions
Analyze processing run costs with a cost analysis tool
Deploying & CI/CD
Code
Synchronization
Leverage an image as code approach for automatic and reproducible builds
Support for both predefined and custom image builds
Synchronize recipes and entire packages
Code can be synched, including Jupyter notebooks, from local to Cloud instances
Geospatial
Analysis
Pre-integration with the most widely used geospatial analysis programs including:
Gamma Software from Gamma Remote Sensing
GDAL, PostgreSQL/PostGIS
Comprehensive Python packages including Xarray and Dask
DEM tools and tiling tools
Leverage the programs you are already using, in the Cloud, without any additional work
Linux
Earth Big Data was built to be an extension of Linux.
Issue commands to deploy, access resources, process recipes, and monitor statuses
Be in complete control from development to operations, without becoming a Cloud expert
Scaling Big Data Analysis
Resource
Scaling
Leverage Cloud compute and storage resources
Deploy resources depending on a job's needs
Automatically launch, hibernate and terminate machines.
Tap into the Amazon Spot Market for cost savings
Set user limits on processing resources
Recipe
Scaling
Create sophisticated recipes that takes advantage of Cloud resources.
Automate the retrieval, pre-processing, and value-added processing of diverse data sources and endpoints
Take advantage of a wide variety of access protocols
Execute recipes as processing scripts
Controlling
Cloud Costs
Use Cloud resources efficiently and economically
Process next to the data to avoid unnecessary transfer costs
Optimize resource configuration
Leverage spot market for reduced costs
Set resource limits
Architecture
Earth Big Data is using AWS components and API tools as its cloud framework. The software has a workflow for recipes, mimics a user's Linux environment and contains modules for geopspatial and remote sensing analysis. The architecture of the software includes:
Administrative tools to manage users and cloud resources
Management tools to maintain user accounts
Creation and management of Cloud machine images
Creation and management of Cloud storage buckets and user access permissions
Creation and management of autoscaling groups
Job management and queuing system
Recipe processor with flexible data sourcing and protocols
Code synchronization for fast CI/CD
Built on Amazon Web Services
Jupyter notebook server on-demand deployment
Credential management
Job monitoring and exception handling
Cost monitoring
GitOps tools and integration with GitHub
REST based API tools
Geospatial processing (add on set of programs)
GAMMA Remote Sensing processing (add on set of programs)
Software Modules
Core
Take advantage of the Cloud for big data analysis without all the complexity. Core provides for the management, deployment, and scaling of applications in the Cloud.
Core + Geospatial
Add integrated tools for geospatial processing, DEM and tiling tools, and Geospatial Python package management.
Core + Geospatial + SAR
Add integration with Gamma Software, from Gamma Remote Sensing, to analyze SAR data leveraging Cloud resources. Support for ISCE software is also available.