ae-dataops

CRITICAL - This skill is self-contained. Use the Global AE CLI Rules below; do not require a separate shared skill for DataOps-side tasks.

The AE Data Development and Operations domain provides capabilities for data warehouse management, flow orchestration, IDE SQL queries, data integration, operations and backfill management, including the following subcommands:

Global AE CLI Rules

AE CLI (ae-cli) is the command-line tool for the AE / TE / ThinkingEngine analysis platform. For AE analysis-side requests, prefer ae-cli and this skill's reference docs over model memory.

Global parameters:

Output and errors:

• Successful commands return machine-readable JSON by default. Envelope may include optional _notice.host_compat.
• Failed commands return { "ok": false, "error": { "type": "...", "message": "...", "hint": "..." } } and exit non-zero.
• CRITICAL — Host compat (do this first): After each ae-cli run, check stderr and _notice.host_compat. If either is present, open the user reply with a short ⚠️ version warning and quote the npm i -g / npx skills add (or update-cluster) lines verbatim, then present the business result. Soft tip; ok: true can still carry the notice.

Safety constraints:

• Read commands can execute directly after required IDs and references are verified.
• Ordinary write commands execute without --yes; use --yes only for a high-risk-write command after explicit user confirmation.
• Never invent command names, flags, JSON payloads, project_id, resource IDs, field names, event names, property names, metric definitions, or date formats. Read the matching command reference and discover real project metadata first.
• NEVER fabricate or guess resource names (reports, dashboards, events, properties, metrics, clusters, tags, alerts). Always use list commands to discover real resources first. If a resource is not found after fuzzy search and full list fallback, explicitly tell the user "resource not found" and stop - do not proceed with fabricated names.

Domains for DataOps: dataops_repo, dataops_datatable, dataops_flow, dataops_operations, dataops_ide, dataops_integration

Core Concepts and Rules

You must understand the following key concepts before use, otherwise errors are highly likely.

ID System

Environment and Defaults

Schema Naming Rules

• DEV environment: ws_${spaceCode}_dev
• PROD environment: ws_${spaceCode}_product

Responsibility Boundaries

Flow Lifecycle

Create DEV Flow → Create/Update DEV SQL, Integration, Workflow Instance Check, or Task Instance Check Tasks → Configure Dependencies/Schedule → Preview Release → Release to PROD → PROD Manual Execution / Operations Troubleshooting

Backfill lifecycle: Discover eligible PROD flow → Create or fully update DRAFT job → Run explicitly → Search / inspect plans → Stop or rerun the complete job; delete only after target inspection

CRON Format (6 fields)

second minute hour day month weekday — Note: one more "second" field than standard 5-field format.

• 0 0 2 * * ? — Daily at 2 AM
• 0 0 */4 * * ? — Every 4 hours
• 0 30 8 * * 1-5 — Weekdays at 8:30

Preset Repository vs Non-Preset Repository

• Preset Repository (te_etl): datasourceId is te_etl@TASK_ENGINE_TRINO, database field is empty, requires gatewayConfig
• Non-Preset Repository: datasourceId is specific datasource ID, database field is required

Scenario Routing

Choose the appropriate scenario skill based on user intent to get complete step-by-step workflow guidance.

1. Space Discovery

dataops_repo exposes only one read command. Use it to discover a valid spaceCode before calling DataOps commands that require one. It returns createTime, spaceCode, and spaceDisplayName.

• If the user already provided a trusted spaceCode, reuse it.
• If spaceCode is unknown, run +list_spaces first.
• If exactly one space is returned, use its spaceCode.
• If multiple spaces are returned and the user intent does not identify one, ask the user which space to use. Do not guess.

# List spaces accessible to the current user
ae-cli dataops_repo +list_spaces

2. Data Table and View Management

Detailed workflow, command flags, examples, and parameter notes live in references/dataops-table.md.

Key constraints:

• Start with dataops_datatable +dict_search_tables for visible DataOps catalog discovery.
• Use dataops_ide +search_tables only for raw engine metadata, and dataops_ide +ide_list_tables only for known catalog/schema browsing.
• Create tables/views with dataops_datatable, not dataops_ide; creation is DEV-only and must be published with +publish_entity.
• DDL follows Trino syntax; current-space view DDL should keep the literal ${env} placeholder.

3. Flow Orchestration

Flow orchestration is divided into two scenario skills: creation and configuration and execution and monitoring.

Lifecycle: DEV configuration and preview → Release to PROD → PROD manual execution and operations troubleshooting

Detailed creation/configuration commands live in references/dataops-flow-create.md. Detailed execution, monitoring, operation instance, task log, and stop commands live in references/dataops-flow-monitor.md. Persistent multi-date backfill jobs live in references/dataops-backfill.md.

Key constraints:

• Create and update tasks in DEV, preview/release before PROD execution.
• Treat +delete_task as high-risk: verify the target with +get_flow_overview, preview with --dry-run, and use --yes only after explicit user confirmation. Deletion affects DEV; release the flow to apply it to PROD.
• +execute_flow always runs PROD; it returns executeId for early stop.
• Prefer flowInstanceId from operations search for stable inspection and troubleshooting.
• A backfill job is persistent and batches multiple base dates; do not emulate it by looping +execute_flow.
• Create and run backfill jobs as separate steps. +rerun_backfill_job reruns the complete job, not only failed plans.
• +update_backfill_job replaces a DRAFT job's complete configuration; inspect the job first and do not treat it as a partial patch. Treat +delete_backfill_job as high-risk and preview it with --dry-run before confirmation.
• Reference workspace parameters in task SQL as ${paramKey}.

4. IDE SQL Queries

Detailed metadata browsing, SQL query, async download, and cancel workflows live in references/dataops-query.md.

Key constraints:

• IDE is query-only; create/modify/delete tables with dataops_datatable.
• Prefer dataops_datatable +dict_search_tables for table discovery unless raw engine metadata or schema browsing is required.
• Submit exactly one read-only SQL query. It creates a platform-bounded download task; rows are not returned inline and the result is not an unlimited or full export.

5. Data Integration

Detailed datasource, metadata browsing, sync solution, execution, and monitoring workflows live in references/dataops-integration.md.

Key constraints:

• Generate sourceConfig, sinkConfig, channelConfig, and fieldsMapping from the reference templates; do not invent keys.
• MySQL Source read partitioning uses sourceConfig.splitColumn; fieldsMapping.shardingKey is column metadata and must not be used for it.
• +save_sync_solution is not a partial patch: call +get_sync_detail --withParams true first, then submit complete configs. syncName is accepted for compatibility but ignored.
• Preset repository sync uses te_etl@TASK_ENGINE_TRINO and requires gateway configuration.
• Use +list_sync_runs to get taskId before stopping a running sync.

Reference Documentation

For detailed command flags and usage, please refer to the command documentation in the references/ directory.