Data

Code Ocean's Data module is a centralised data asset management solution designed for computational scientists, R&D teams, and life sciences organisations working in the cloud. It provides a single place to manage, access, and share data from virtually any source — whether stored natively within Code Ocean or kept in external systems — while automatically tracking lineage, ensuring reproducibility, and eliminating unnecessary data duplication.

Data is purpose-built for computational science workflows, integrating seamlessly with the broader Code Ocean platform including Compute Capsules, Pipelines, Lineage Graph, Collections, and the API. It is designed to meet all FAIR principles and supports organisations of any scale, with no limits on data size.

Data Source Support and Storage

• Native integrations with AWS S3, AWS Athena, and Databricks for bringing in data from common cloud sources.
• Local upload supported for direct ingestion of data into the platform.
• Additional data sources accessible via custom Data Connector Capsules available in Code Ocean Apps, or through the API.
• Internal data brought into Code Ocean is stored in an organised S3 bucket within your own AWS account for enhanced reproducibility and performance.
• External data can remain in place and be accessed by linking to S3 bucket locations, offering flexibility without requiring data migration.
• All recently used internal data is cached to EFS for fast access during computational work.

Key Capabilities

• Multi-source data management: Use data from multiple sources simultaneously in computational work, with the choice to ingest data for reproducibility or keep it external for convenience.
• Instant attachment to analyses and pipelines: Data assets are managed independently from Compute Capsules and Pipelines, allowing a single Data asset to be used across multiple Capsules, Pipelines, and by multiple users simultaneously without duplication.
• Sharing and access control: All Data assets can be shared with other users and groups within your AWS deployment, with built-in permission and access control. User Secrets and assumable IAM roles ensure secure access to private data.
• Result provenance tracking: Click into any Result Data to view its full provenance via the Lineage Graph, showing all source data, Compute Capsules, and Pipelines involved in generating it.
• Custom metadata: Admins can create custom metadata keys with specified value and input types, enabling improved search, management, and organisation of data assets. Controlled vocabularies ensure consistency across the organisation.
• API-triggered automation: Newly uploaded data can trigger upstream processing pipelines automatically, enabling preparation and delivery of data to defined groups or individuals without manual intervention.

Platform Integration and Workflow

• Collections: Group and organise Data assets alongside Compute Capsules, Pipelines, and Apps within Collections to keep all project-related assets visible and findable in one place.
• Lineage Graph: Automatically tracks all Compute Capsules, Pipelines, and source Data used to generate any result, providing an immutable record of data provenance.
• API: Enables programmatic access and event-driven automation, such as triggering pipelines when new data is uploaded.

Computational Science Use Cases

• Data analysis: Use ready-made template Compute Capsules or develop custom workflows in your preferred language and IDE, with built-in containerisation to guarantee reproducibility.
• Data management: Manage organisational data with access controls, custom metadata, and controlled vocabularies, built to meet all FAIR principles.
• Bioinformatics pipelines: Build and monitor pipelines using a visual builder, or import from nf-core in one click. Runs on AWS Batch out-of-the-box for automatic scaling with no setup required.
• ML model development: Supports AI, ML, Deep Learning, and Generative AI workflows with GPU-ready environments, MLflow integration for model tracking and lifecycle management, and out-of-the-box reproducibility and lineage.
• Multiomics: Efficiently analyse large multimodal datasets using scalable compute and storage, cached R and Python packages, and preloaded multiomics analysis software.
• Imaging: Process images ranging from individual files to petabyte-scale datasets using dedicated desktop applications or custom deep learning pipelines, with no DevOps required.
• Data and model provenance: Automated lineage graph generation provides a visual representation of every Capsule, Pipeline, and Data asset involved in any computation.

Code Ocean's Data module is deployed within your own AWS Virtual Private Cloud (VPC), ensuring data sovereignty and security. Cloud cost management features — including spot instances, idleness detection, and automated shutdown — help organisations control compute expenditure while scaling to meet research demands.