Skip to main content

Setting up Spring Cloud Data Flow Server

Spring cloud data flow server can be used to setup the pipeline of tasks or streams. Tasks and streams can be registered with data flow server and can be executed or monitored either using cloud data flow shell or data flow server UI console/ dashboard.

Setting up Data flow server

Spring cloud data flow server can be setup using two ways, either use spring provided spring boot application or develop your own spring cloud data flow server as spring boot application.

Using Spring provided Data flow server application

There are many ways to setup data flow server as given below.
  • Local Machine
  • Cloud Foundry
  • Kubernetes
Please refer below documentation for more details for above installation types.
https://dataflow.spring.io/docs/installation/

Create Spring boot application as Data flow server

Here I will show you how to create your own spring boot application as spring cloud data flow server which you can run anywhere locally or in cloud. It gives you more freedom with application to setup the other dependencies like database or security features.

Maven Dependencies

We have used Spring Boot version 2.1.6.RELEASE which is supported by the current version of data flow server (2.2.1.RELEASE). Below are the dependencies required in pom.xml
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-dataflow-server</artifactId>
            <version>2.2.1.RELEASE</version>
        </dependency>

application.properties

In this properties file we are just creating an in-memory database using H2. However it is not required as it will automatically setup one if we don't configure it. We can use these properties to use any other database.
spring.datasource.url=jdbc:h2:tcp://localhost:19092/mem:dataflow
spring.datasource.jdbcUrl=jdbc:h2:tcp://localhost:19092/mem:dataflow
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=

spring.jpa.database-platform=org.hibernate.dialect.H2Dialect

MainDataFlowServerApplication.java

This is our main class which need to be annotated  with @EnableDataFlowServer annotation. Since we are going to run this application in our local machine, we need to exclude some auto configurations like cloud foundry, kubernetes etc. until we intend to actually configure them. When required we can remove them from exclusions and configure necessary properties for auto configuration. See the below code.
@EnableDataFlowServer
@SpringBootApplication(
        exclude = {SessionAutoConfiguration.class,
                ManagementWebSecurityAutoConfiguration.class,
                SecurityAutoConfiguration.class,
                UserDetailsServiceAutoConfiguration.class,
                LocalDeployerAutoConfiguration.class,
                CloudFoundryDeployerAutoConfiguration.class,
                KubernetesAutoConfiguration.class}
)
public class MainDataFlowServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(MainDataFlowServerApplication.class, args);
    }
}

Git URL

Below GIT URL contains complete source code.
https://github.com/thetechnojournals/spring-tutorials/tree/master/SpringCloudDataFlowServer

Running data flow server application

Once you done with all required code and configuration setup, you can build the application and use below maven command to start your data flow server.
mvn spring-boot:run
Once application is started, hit the below URL in browser and you will see the given screen as dashboard of data flow server application where you can see all registered task and streams.\
URL: http://localhost:9393/dashboard/

spring-cloud-dataflow-server

Check my another post on registering and executing Spring Batch application with Spring cloud data flow server.
https://www.thetechnojournals.com/2019/12/spring-batch-job-with-spring-cloud-data.html
Labels:

Comments

Post a Comment

Popular Posts

Setting up kerberos in Mac OS X

Kerberos in MAC OS X Kerberos authentication allows the computers in same domain network to authenticate certain services with prompting the user for credentials. MAC OS X comes with Heimdal Kerberos which is an alternate implementation of the kerberos and uses LDAP as identity management database. Here we are going to learn how to setup a kerberos on MAC OS X which we will configure latter in our application. Installing Kerberos In MAC we can use Homebrew for installing any software package. Homebrew makes it very easy to install the kerberos by just executing a simple command as given below. brew install krb5 Once installation is complete, we need to set the below export commands in user's profile which will make the kerberos utility commands and compiler available to execute from anywhere. Open user's bash profile: vi ~/.bash_profile Add below lines: export PATH=/usr/local/opt/krb5/bin:$PATH export PATH=/usr/local/opt/krb5/sbin:$PATH export LDFLAGS=&
85 comments
Read more

Why HashMap key should be immutable in java

HashMap is used to store the data in key, value pair where key is unique and value can be store or retrieve using the key. Any class can be a candidate for the map key if it follows below rules. 1. Overrides hashcode() and equals() method.   Map stores the data using hashcode() and equals() method from key. To store a value against a given key, map first calls key's hashcode() and then uses it to calculate the index position in backed array by applying some hashing function. For each index position it has a bucket which is a LinkedList and changed to Node from java 8. Then it will iterate through all the element and will check the equality with key by calling it's equals() method if a match is found, it will update the value with the new value otherwise it will add the new entry with given key and value. In the same way it check for the existing key when get() is called. If it finds a match for given key in the bucket with given hashcode(), it will return the value other
1 comment
Read more

Entity to DTO conversion in Java using Jackson

It's very common to have the DTO class for a given entity in any application. When persisting data, we use entity objects and when we need to provide the data to end user/application we use DTO class. Due to this we may need to have similar properties on DTO class as we have in our Entity class and to share the data we populate DTO objects using entity objects. To do this we may need to call getter on entity and then setter on DTO for the same data which increases number of code line. Also if number of DTOs are high then we need to write lot of code to just get and set the values or vice-versa. To overcome this problem we are going to use Jackson API and will see how to do it with minimal code only. Maven dependency <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.9.9</version> </dependency> Entity class Below is
Post a Comment
Read more

Multiple data source with Spring boot, batch and cloud task

Here we will see how we can configure different datasource for application and batch. By default, Spring batch stores the job details and execution details in database. If separate data source is not configured for spring batch then it will use the available data source in your application if configured and create batch related tables there. Which may be the unwanted burden on application database and we would like to configure separate database for spring batch. To overcome this situation we will configure the different datasource for spring batch using in-memory database, since we don't want to store batch job details permanently. Other thing is the configuration of  spring cloud task in case of multiple datasource and it must point to the same data source which is pointed by spring batch. In below sections, we will se how to configure application, batch and cloud task related data sources. Application Data Source Define the data source in application properties or yml con
47 comments
Read more