Spring Batch integration module for GridGain

For the purpose of using Spring Batch in a scalable and distributed manner to process huge amount of data, I am actually developing some components to make integration of Spring Batch with compute/data grid easier.

Different solutions is offered by Spring Batch to provide scalability, the one that best suit my needs is remote chunking.

As I already done some investigation before using GridGain I chose this framework to implement a distributed remote chunking system that can be easily integrated into any existing Spring Batch systems.

Using GridGain is really straightforward, and setting up a grid on a development machine doesn't need so much configuration.

The only issue I faced is due to the fact that GridGain use serialization to deploy tasks on nodes, in order to be able to deploy a remote ChunkProcessor, it must contains serializable ItemProcessor and ItemWriter, which unfortunately is not the case by default.

So instead of creating new interfaces, I made a SerializableChunkProcessor which only accept serializable ItemProcessor and ItemWriter. It's surely not the smarter solution, but since I can't modify default interfaces in Spring Batch and I don't want to create my own interfaces, this workaround will suffice.

Usage
Here is the job application context used for the integration test, as you can see the 'real' ItemProcessor / ItemWriter are injected into the GridGain chunk writer:
<!-- Standards items component --><br/> <bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader"><br/> <property name="resource" value="classpath:sentences.txt"/><br/> <property name="lineMapper"><br/> <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper"><br/> <property name="fieldSetMapper"><br/> <bean class="org.springframework.batch.integration.chunk.gridgain.SentenceFieldSetMapper"/><br/> </property><br/> <property name="lineTokenizer"><br/> <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer"><br/> <property name="delimiter" value="\"/><br/> </bean><br/> </property><br/> </bean><br/> </property><br/> </bean><br/> <br/> <bean id="processor" class="org.springframework.batch.integration.chunk.gridgain.WordCounterProcessor"/><br/> <br/> <bean id="writer" class="org.springframework.batch.integration.chunk.gridgain.ConsoleWriter"/><br/> <br/> <!-- Chunk Writer --><br/> <bean id="chunkWriter" class="org.springframework.batch.integration.chunk.gridgain.ChunkGridGainItemWriter"><br/> <property name="chunkProcessor"><br/> <bean class="org.springframework.batch.integration.chunk.SerializableChunkProcessor"><br/> <property name="itemProcessor" ref="processor"/><br/> <property name="itemWriter" ref="writer"/><br/> </bean><br/> </property><br/> <property name="grid" ref="grid"/><br/> <property name="throttleLimit" value="16"/><br/> </bean><br/> <br/> <!-- Job --><br/> <job id="gridgainJob" xmlns="http://www.springframework.org/schema/batch"><br/> <step id="step1"><br/> <tasklet><br/> <chunk reader="reader" writer="chunkWriter" commit-interval="4"/><br/> </tasklet><br/> </step><br/> </job>
Download
You can download the spring-batch-integration-gridgain module here:
http://github.com/downloads/aloiscochard/spring-batch-integration-gridgain/spring-batch-integration-gridgain-0.0.1-SNAPSHOT.jar

If you want to see a full working sample, take a look at the integration test. The full project sources can be downloaded here:
http://github.com/aloiscochard/spring-batch-integration-gridgain