watch out for those big transactions and solr

I have a quartz job that will periodically call a grails service to tell it import data located within csv files.  For reasons unknown, the jobs were failing with the following stacktrace (which really didn't tell me anything)

Apr 24, 2010 5:19:45 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=1} status=0 QTime=34 
Apr 24, 2010 5:19:45 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
	commit{dir=/usr/local/apache-solr/apache-solr-1.4.0/georepo/solr/data/index,segFN=segments_5507,version=1262051438693,generation=239767,filenames=[_5lr8.prx, _5lr9.fdt, _5lr8.fdt, _5lr4.tii, _5lr5.prx, _5lrc.prx, _5lr9.fdx, _5lr8.fdx, _5lr6.fdx, _5lr8.tis, _5lr9.tis, _5lr6.tis, _5lr6.fdt, _5lr7.frq, _5lr9.tii, segments_5507, _5lr6.tii, _5lr8.tii, _5lr4.prx, _5lr7.fnm, _5lr9.fnm, _5lrb.fdt, _5lr9.prx, _5lrc.fnm, _5lr8.fnm, _5lr9.frq, _5lrb.prx, _5lr4.nrm, _5lrc.frq, _5lrb.nrm, _5lr7.nrm, _5lr9_3.del, _5lr8_2.del, _5lra.frq, _5lr4.tis, _5lr6_2.del, _5lr7_3.del, _5lrb.fdx, _5lra.fnm, _5lr5.fnm, _5lr6.frq, _5lra.fdx, _5lr9.nrm, _5lrb.tii, _5lr5_3.del, _5lra_2.del, _5lr5.frq, _5lr8.frq, _5lr4.frq, _5lr4.fdt, _5lr7.fdt, _5lr5.tii, _5lrb_2.del, _5lr4.fdx, _5lr7.fdx, _5lr6.fnm, _5lra.prx, _5lra.fdt, _5lr7.prx, _5lrb.tis, _5lrb.fnm, _5lrc.tii, _5lr5.nrm, _5lr5.tis, _5lra.tis, _5lr5.fdt, _5lr5.fdx, _5lrc.fdt, _5lr6.nrm, _5lr4.fnm, _5lra.nrm, _5lr7.tis, _5lr4_2.del, _5lr7.tii, _5lrc.fdx, _5lrc.tis, _5lr8.nrm, _5lrc.nrm, _5lrb.frq, _5lra.tii, _5lr6.prx]
Apr 24, 2010 5:19:45 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1262051438693
Apr 24, 2010 5:19:45 PM org.apache.solr.update.processor.LogUpdateProcessor finish
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:923)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:880)
	

Big help here!

Anyway...figured it out.

My csvfile service is just that, a service, and as such, the records that were being inserted into the database were part of a large transaction. A second service that I had that would process the records and update the solr index.  This was probably on the order of a 2k records at once, causing the solr call to fail.

For now, I am going to move the csvfileservice to not be a service, but loop through the csvfile records and commit each one at a time.  This way, the database can be updated successfully (since the solr call failed, the db commit did not happen), and the solr index can be incrementally updated, and WORK!