Creating a DataStage flow

DataStage flows are the design-time assets that contain data integration logic.

You can create a blank DataStage flow and add connectors and stages to it or you can import an existing DataStage flow from an ISX file.

The basic building blocks of a flow are:
  • Stages that connect to data sources to read or write files and to process data.
  • Links that connect the stages along which your data flows.

Palette and canvas in IBM DataStage

DataStage flows and their associated objects are organized in projects. To start, open an existing project or create a new project.

Creating a DataStage flow by individually adding connectors and stages

To create a DataStage flow by individually adding connectors and stages, complete the following steps.

  1. Open an existing project or create a project.
  2. Click Add to project + and select DataStage flow or select New DataStage Flow + in the DataStage flows section of the page.
  3. Add a name and optional description for the new flow on the New tab of the New DataStage flow page.
  4. Drag connectors or stages from the palette onto the canvas as nodes and arrange them as you like. Connect these nodes on the canvas by clicking the arrow icon on a node and dragging it to the node you want to connect to.

    This action creates a link between the nodes.

    Note: The connections that you add to the flow must be created already in the project that you are working in. For more information, see Adding connections to projects.
  5. Double-click a node to open up its Details card, where you can specify configurations and settings for the node.
  6. Click Run when you are done setting up the flow.

    The flow is automatically saved, compiled into a job, and run. You can view logs for both the compilation and job run.

After the flow is compiled into a job, you can re-run the job, set a schedule, monitor the job, and update the environment that you want to run it in. For more information about running jobs, see Creating, scheduling, and running jobs. For more information about updating the DataStage environment where you want your jobs to run, see DataStage environments.

Editing a DataStage flow

You can use the following actions to edit a DataStage flow.

  • Drag a stage or connector and drop it on a link between two nodes that are already on the canvas. Links are automatically added for the new node and columns are automatically propagated. Click Run again to see the results.
  • Manually detach and reattach links from nodes on the canvas by hovering your pointer over them and clicking the end points of the links.
  • Drag a stage or connector from the palette and drop it onto a link that is already on the canvas. The stage or connector is automatically linked to the node on either side of it and the columns in the DataStage flow automatically propagated.

Learn more

Examples

Creating a DataStage flow

Watch the following video for an example of how to create a simple DataStage flow.

This video provides a visual method as an alternative to following the written steps in this documentation.

Importing a DataStage flow into a project

Watch the following video for an example of how to import a DataStage flow into a project.

This video provides a visual method as an alternative to following the written steps in this documentation.