c# - Which technology to use for running crawler and updating database in asp.net website? -

-


c# - Which technology to use for running crawler and updating database in asp.net website? -

i developing project college , need suggestions on development. website shows info other websites links, images etc.

i have prepared below given model website.

a home.aspx page shows info tables (sql server).

i have coded crawler (in c#) can crawl (fetch data) required website data.

i want way through can run crawler @ end time interval , can insert updates in tables. want can updated info in database home.aspx shows updated info. (its smaller version of google news website)

i want host wesbite in shared hosted environment (i.e 3rd party hosting provider company , may utilize iis platform)

i posted simliar situation different .net forums , communities , suggested lot of different things such as

create web service (is necessary ?)

use wcf

create console application , run windows task sheduler (is okay asp.net (win forms website) , in shared hosted)

run crawler on local machine , update database accordingly. (no want online) etc etc

please suggest me clear way out finish task. please suggest elobrated technology , methods suits project.

waiting...

thanks...

your shared host constraint impacts on technologies restrictions.

in theory, best way host crawler have been windows service, since can take advantage of windows services configuration. service up, can automatically started @ startup, writes errors in event log, can automatically restarted after failure...

then, home.aspx have been regular website in iis.

if remain on shared host (where cannot setup service), have create crawler module run on application startup.

problem is, iis application pool doesnt live forever if web site not in use, , may stop crawler. configurable, dont know how much in shared host.

in iis 7.5, think starting module @ application warm up

finally if need run crawler @ interval times (like every day @ midnight), if shared host not allow set task scheduling, think quartz framework, allow perform task scheduling within application (without intervention of os)

c# asp.net database wcf

Comments

Post a Comment

Popular posts from this blog

How do I check if an insert was successful with MySQLdb in Python? -

-
How do I check if an insert was successful with MySQLdb in Python? - i have code: cursor = conn.cursor() cursor.execute(("insert new_files (videos_id, filename, " "is_processing) values (%s,%s,1)"), (id, filename)) logging.warn("%d", cursor.rowcount) if (cursor.rowcount == 1): logging.info("inserted values %d, %s", id, filename) else: logging.warn("failed insert values %d, %s", id, filename) cursor.close() fun is, cursor.rowcount always one, though updated database create videos_id unique key. is, insert fails because in tests same videos_id going appear (and when check database, nil inserted). whatever reason, rowcount 1 - logging.warn have spits out rowcount of 1. so, question: can utilize rowcount work out if insert went fine? if so, (presumably) doing wrong? otherwise, how check if insert went fine? your code not commit after modifications (your modifications rolled back). should...
Read more

delphi - blogger via idHTTP : error 400 bad request -

-
delphi - blogger via idHTTP : error 400 bad request - i'm trying post blogger using idhttp component, however, i'm getting "http/1.0 400 bad request" error. procedure tform1.button1click(sender: tobject); var request,response,req : tstringlist; auth,blogid : string; begin blogid := '00000000000000000000000'; request := tstringlist.create; response := tstringlist.create; req := tstringlist.create; idhttp1.request.connection := 'keep-alive'; idhttp1.request.contenttype := 'application/x-www-form-urlencoded'; idssliohandlersocket1.ssloptions.method := sslvsslv23; request.clear(); request.values['accounttype'] := 'google'; request.values['email'] := 'xxx@gmail.com'; request.values['passwd'] := 'yyy'; request.values['service'] := 'blogger'; response.text :=idhttp1.post('https://www.google.com/accounts/clientlogin',request); auth := resp...
Read more

postgresql - ERROR: operator is not unique: unknown + unknown -

-
postgresql - ERROR: operator is not unique: unknown + unknown - i have postgis database , have compute new values rows in new column. these values should average of values of several columns. query: insert bdps (da_m) values (avg('da_1'+'da_2'+'da_3'+'da_4'+'da_5'+'da_6'+'da_7')); in query bdps database, da_m new column , da_1 da_7 existing columns have double precision type. da_m created using alter table bdps add together column da_m double precision; i next error: error: operator not unique: unknown + unknown line 2: values (avg('da_1'+'da_2'+'da_3'+'da_4'+'da_5'+'da_6'+'da_7... ^ hint: not take best candidate operator. might need add together explicit type casts. ********** error ********** error: operator not unique: unknown + unknown sql state: 42725 hint: not take best candidate operator. might need add togethe...
Read more