Enhancing the Efficiency of Smart Crawler

Abstract

Now days, as the data in the internet is increasing and we are dumping the information in the web, some of the major problems heads towards us is that managing the big data, performance of the web, including these one more causes a serious aspect that is accessing the deep web services with high performance and the performance of the web can be measured by wspt tool (web server performance testing tool). The main purpose of wspt is to measure the performance of server side application. So to overcome the problem of accessing the deep web services hence proposed a system called as smart crawler and WPC, FSC, FCC and also the major constraint here is that maintaining the big data of the hidden web services is again a challenging problem because sometimes in deep web services the user requires only some highly ranked or visited sites but the existing smart crawler lists all the sites unrelated of ranks by this the user will be uncomfortable with it, so hence proposed the different framework that that can overcome this problem that contains the two kinds of database 1) temporary – contains only the highly ranked sites 2) permanent – contains all the related links of the search string given by the user.