How to terminate process with Lucene NRT Reader/ Writer gracefully? -

-


How to terminate process with Lucene NRT Reader/ Writer gracefully? -

we using lucene's near real-time search feature full-text search in our application. since commits costly, commit index after every 10 documents added (we expect around 150 200 documents per hr indexing). now, if want terminate process, how create sure documents in memory committed disk before process killed? there approach recommended here? or document volume less bother , should commit on every addition?

should track uncommitted documents? , if process gets killed before committed disk, should index these uncommitted ones 1 time again when process start up?

lucene nrt used in process runs embedded jetty. right approach send shutdown command (invoke servlet) jetty , wait till documents committed , terminate using system.exit()?

you add together hook commit buffered documents in destroy method of servlet , create sure embedded servlet container shut downwards before calling system.exit (maybe adding shutdown hook jvm).

but still not perfect. if process gets killed, buffered info lost. solution utilize soft commits. soft commits inexpensive commits (no fsync performed) no info lost if process gets killed (but info still lost if server shuts downwards unexpectedly).

to sum up:

shutdown hook best throughput data can lost if process gets killed soft commit no info lost if process gets killed data may lost if server shuts downwards unexpectedly hard commit (default) no info loss @ all slow (need perform fsync)

lucene lucene-nrt

Comments

Post a Comment

Popular posts from this blog

How do I check if an insert was successful with MySQLdb in Python? -

-
How do I check if an insert was successful with MySQLdb in Python? - i have code: cursor = conn.cursor() cursor.execute(("insert new_files (videos_id, filename, " "is_processing) values (%s,%s,1)"), (id, filename)) logging.warn("%d", cursor.rowcount) if (cursor.rowcount == 1): logging.info("inserted values %d, %s", id, filename) else: logging.warn("failed insert values %d, %s", id, filename) cursor.close() fun is, cursor.rowcount always one, though updated database create videos_id unique key. is, insert fails because in tests same videos_id going appear (and when check database, nil inserted). whatever reason, rowcount 1 - logging.warn have spits out rowcount of 1. so, question: can utilize rowcount work out if insert went fine? if so, (presumably) doing wrong? otherwise, how check if insert went fine? your code not commit after modifications (your modifications rolled back). should...
Read more

delphi - blogger via idHTTP : error 400 bad request -

-
delphi - blogger via idHTTP : error 400 bad request - i'm trying post blogger using idhttp component, however, i'm getting "http/1.0 400 bad request" error. procedure tform1.button1click(sender: tobject); var request,response,req : tstringlist; auth,blogid : string; begin blogid := '00000000000000000000000'; request := tstringlist.create; response := tstringlist.create; req := tstringlist.create; idhttp1.request.connection := 'keep-alive'; idhttp1.request.contenttype := 'application/x-www-form-urlencoded'; idssliohandlersocket1.ssloptions.method := sslvsslv23; request.clear(); request.values['accounttype'] := 'google'; request.values['email'] := 'xxx@gmail.com'; request.values['passwd'] := 'yyy'; request.values['service'] := 'blogger'; response.text :=idhttp1.post('https://www.google.com/accounts/clientlogin',request); auth := resp...
Read more

postgresql - ERROR: operator is not unique: unknown + unknown -

-
postgresql - ERROR: operator is not unique: unknown + unknown - i have postgis database , have compute new values rows in new column. these values should average of values of several columns. query: insert bdps (da_m) values (avg('da_1'+'da_2'+'da_3'+'da_4'+'da_5'+'da_6'+'da_7')); in query bdps database, da_m new column , da_1 da_7 existing columns have double precision type. da_m created using alter table bdps add together column da_m double precision; i next error: error: operator not unique: unknown + unknown line 2: values (avg('da_1'+'da_2'+'da_3'+'da_4'+'da_5'+'da_6'+'da_7... ^ hint: not take best candidate operator. might need add together explicit type casts. ********** error ********** error: operator not unique: unknown + unknown sql state: 42725 hint: not take best candidate operator. might need add togethe...
Read more