Open Scala IDE - Create your own workspace — Help - Eclipse Marketplace - search for “Gradle IDE pack 3.8.x + 1.0.x” (note: my IDE version is Neon, hence this Gradle version is compatible). I have a Scala project I share via git between two (windows) machines. I have them set up using SBT and sbt-eclipse so I can edit and test within eclipse or build and test from the command-line. Unfortunately my user name (and, therefore, the user profile directory) is different on the different machines. Apache spark - a very known in memory computing engine to process big data workloads. Scala IDE(an eclipse project) can be used to develop spark application. The main agenda of this post is to setup development environment for spark application in scala IDE and run word count example.
As a coder, we are more comfortable with the editor tools (specially Eclipse IDE) for rapid development , build & continuous integration. When first time I was trying to develop some Kafka producer and consumer using Scala, I was wondering if I could setup the same through eclipse to make life easier, however after a lot of hit and trial finally able to setup and successfully executed my first program.
Likewise, some other friends out there might struggling on the same. Hence thought to share the simple steps to get the Kafka setup and run program from eclipse IDE done.
Important : Before proceed, please ensure you zookeeper & Kafka server already running. If not the Mac users can always refer Step By Step : Install Kafka in OSX / macOS Sierra for setting up and running ZK & Kafka server
Step 1: Instal Scala IDE (Eclipse for scala)
Download Scala IDE from the official site http://scala-ide.org
[Please ensure the IDE version you are downloading is meeting the system requirement]
Step 2: Install Gradle in the downloaded Scala IDE
Open Scala IDE -> Create your own workspace —> Help -> Eclipse Marketplace -> search for “Gradle IDE pack 3.8.x + 1.0.x” (note: my IDE version is Neon, hence this Gradle version is compatible)
Step 4: Importing Kafka libraries into eclipse workspace
- Download the Kafka latest build from git https://github.com/sreev/kafka/tree/0.8.2
- Download the zip into Desktop (to local system)
- Extract the downloaded zip
- Go to eclipse IDE -> File -> Import -> Gradle (STS) project -> Next -> Brouse the extracted Kafka directory from you local disk -> click 'Build Model' -> next -> Finish
The following are the step by step screenshots for importing Kafka project
Step 5: Now expand the project “example” -> run the “KafkaConsumerProducerDemo” class as java program
Now you should able to see the producer and consumer running in your eclipse console and printing the data in consumer console those are published by the producer.
Kafka properties for the above, where we have set all the required configuration params
Do task chair. Now you can also observe the topic name (you have provided in your program) from your terminal where the kafka server is already running.
Eclipse Scala Search
Conclusion : And you all set for writing more complex kafka code and executing from eclipse IDE (can be Scala or Java as per the need)