Open Source Options

1. Full Text Search
   1. http://en.wikipedia.org/wiki/Full_text_search#Software
   2. http://www.mediawiki.org/wiki/Fulltext_search_engines
2. Interesting Search in My Opinion
   1. Sphinx
   2. MySql Search
   3. Sql Server search
   4. Lucene and Elastic Search on top of Lucene.
3. Full Text search comparison
   1. http://full-text-search.findthebest.com/
   2. 

      http://wrg.upf.edu/WRG/dctos/Middleton-Baeza.pdf
   3. A very nice comparison http://zooie.wordpress.com/2009/07/06/a-comparison-of-open-source-search-engines-and-indexing-twitter/
   4. http://taschenorakel.de/mathias/2012/04/18/fulltext-search-benchmarks/
   5. http://www.dbbest.com/blog/lucene-vs-sql-server-fts/
   6. http://beerpla.net/2009/09/03/comparison-between-solr-and-sphinx-search-servers-solr-vs-sphinx-fight/
4. Sphinx http://sphinxsearch.com/
   1. I have personally used Sphinx with a Ruby on Rails project by installing sphinx in the background, installing a gem to interact with sphinx, defining which attributes in the model file to index, how to do searching, using delta index to speed up the process by using a delayed gem which would make a local copy of the change and when the index was updated after some period sometimes days sometimes after a week (done through cron job) then it would move the delta changes to the full index. I found the sphinx server easy to use once i got the hang of it. The delta index is normally smaller then the full index and is often the most recent changes which have not been integrated into the full index. This is normally used to avoid updating the whole index(as re-indexing is a time consuming process which can take long time based on the index size). 
   2. http://en.wikipedia.org/wiki/Sphinx_(search_engine)
   3. Can be used as stand-alone or with  MySQL, MariaDB and PostgreSQL, or by using ODBC with ODBC-compliant DBMS’s
   4. Sphinx latest release download http://sphinxsearch.com/downloads/release/
   5. Documentation 
      

      http://sphinxsearch.com/docs/2.1.3/
   6. Support for many programming languages integration and highly scalable.
   7. Has a lot of features related to natural language processing like using stopwords, tokenization etc.
   8. Note that the original contents of the fields are not stored in the Sphinx index. The text that you send to Sphinx gets processed, and a full-text index (a special data structure that enables quick searches for a keyword) gets built from that text. But the original text contents are then simply discarded. Sphinx assumes that you store those contents elsewhere anyway
   9. There are multiple modes of searching which can be found
      

      here
   10. http://stackoverflow.com/questions/737275/comparison-of-full-text-search-engine-lucene-sphinx-postgresql-mysql
5. MySql Full Text Search 
   

   http://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html
   1. Modes of search:
      1. A boolean search interprets the search string using the rules of a special query language
      2. A natural language search interprets the search string as a phrase in natural human language
      3. A query expansion search is a modification of a natural language search
6. Sql Server Full Text Search http://technet.microsoft.com/en-us/library/ms142571.aspx
   1. The beginning of the article give overview of text search, functionality, architecture, and modes of searching.
   2. Interesting section on this page are the related tasks at the end which gives more detail on how exactly to do the search. The most helpful article is the first one on how to get started with full text search http://technet.microsoft.com/en-us/library/ms142497.aspx
7. http://lucene.apache.org/solr/ Apache Solr/Lucene
   1. Rest Api
   2. Stand alone
   3. Tutorial http://lucene.apache.org/solr/4_6_0/tutorial.html
8. Interesting Project on top of Lucene http://www.elasticsearch.org/overview/
   1. Interesting because it supports real time analytics and real time search, document oriented, restful(like lucene) and full text search
9. BaseX http://basex.org/
   1. Xml Database with full text search using XPath for search.
10. Datapark search http://www.dataparksearch.org/ for search within a website or group or intarnet
    1. Documentation http://www.dataparksearch.org/index.en.html
11. ht://Dig http://www.htdig.org/
12. Apache Lucy http://lucy.apache.org/
    1. Loose C port for Lucene(Java search engine)
    2. Full Text Search
13. Lemur Project http://www.lemurproject.org/
14. Search for Websites http://www.searchdaimon.com/
15. http://swish-e.org/ Swish-e

Venture Capital – Startup Network

• Original Documents:

  • Initial Report PDF
    

    Report
  • Initial input data file Excel File( 
    

    VCdeals
    ), CSV File( 
    

    VCdeals
    )
  • git repo /home/ec2-user/GitRepo/R/VentureCapital.git/ ec2:/home/ec2-user/GitRepo/R/VentureCapital.git ec2: is a alias

  • Clustering

    • For Firms
      

      FirmNodes.csv
      

      FirmEdges.csv
      

      FirmIsolates.csv
      

      FirmEdgesUpdatedWithCluster.csv
      FirmCluster.json
    • For Companies
      

      CompanyNodes.csv
      

      CompanyIsolates.csv
      

      CompanyEdges.csv

  • Command to import data:

    • Change directory to mongo bin path.
    • ./mongoimport -d ndataconsulting -c VCDeals –type csv –file /home/ntreees/VCdeals.csv –headerline
    • This above command import the csv file to ndataconsulting database and within that uses VCDeals collection.
    • Indexes single column:
      • db.VCDeals.ensureIndex({“companyname”:1})
      • db.VCDeals.ensureIndex({“datefund”:1})
      • db.VCDeals.ensureIndex({“firmname”:1})
      • db.VCDeals.ensureIndex({“companysituation”:1})
      • db.VCDeals.ensureIndex({“companypublicstatus”:1})
      • db.VCDeals.ensureIndex({“companystatecode”:1})

  • iGraph R Module

    • http://igraph.wikidot.com/community-detection-in-r
    • IGraph Community Detection Details http://www.r-bloggers.com/summary-of-community-detection-algorithms-in-igraph-0-6/
    • iGraph Documentation (
      

      http://igraph.sourceforge.net/doc/R/00Index.html
      )
    • iGraph Tutorial (http://igraph.sourceforge.net/igraphbook/igraphbook-datamodel.html)
    • Drawing Graph (http://horicky.blogspot.com/2012/04/basic-graph-analytics-using-igraph.html)
    • Psuedo Inverse(
      

      http://hosho.ees.hokudai.ac.jp/~kubo/Rdoc/library/corpcor/html/pseudoinverse.html
      )
    • Get Adjency Graph(http://stackoverflow.com/questions/14849835/how-to-calculate-adjacency-matrices-in-r)
    • layout=layout.fruchterman.reingold Force Based Implementation
      

      http://igraph.sourceforge.net/doc/R/plot.common.html
    • Drawing Graph
      

      http://igraph.sourceforge.net/doc/R/plot.common.html
    • Degree of Graph
      

      http://igraph.sourceforge.net/doc/R/degree.html
    • Laplacian of Graph
      

      http://igraph.sourceforge.net/doc/R/graph.laplacian.html
    • Example to create graph from data http://igraph.sourceforge.net/igraphbook/igraphbook-creating.html
    • Integration With R
      • R Chart With Rickshaw
        

        http://timelyportfolio.github.io/rCharts_rickshaw_gettingstarted/
      • http://vis.supstat.com/2012/11/contour-plots-with-d3-and-r/
      • http://code.shutterstock.com/rickshaw/tutorial/introduction.html
      • https://groups.google.com/forum/#!topic/d3-js/E5Bcl7M2HUw
      • http://civilstat.com/?p=536
      • RMongo http://cran.r-project.org/web/packages/RMongo/index.html
    • Interesting Snippets
      • http://www.r-bloggers.com/network-visualization-in-r-with-the-igraph-package/
      • http://markov.uc3m.es/2012/11/temporal-networks-with-igraph-and-r-with-20-lines-of-code/
      • http://rdatamining.wordpress.com/2012/05/17/an-example-of-social-network-analysis-with-r-using-package-igraph/
      • http://nsaunders.wordpress.com/2010/04/21/experiments-with-igraph/
      • http://stackoverflow.com/questions/9876267/r-igraph-community-detection-edge-betweenness-method-count-list-members-of-e
      • http://rulesofreason.wordpress.com/2012/11/05/network-visualization-in-r-with-the-igraph-package/
      • http://somelab.net/2012/11/how-to-create-a-network-animation-with-r-and-the-igraph-package/
      • 

        http://cneurocvs.rmki.kfki.hu/igraph/doc/R/plot.common.html

  • Detail about the algorithm (Newman-Girvan cohesion-based clustering algorithm) :

    • Is a algorithm which is used to find Community Structure. For a community structure n0rmally a set of nodes are densely connected to each other form a group and these groups are sparsely connected to other groups. Basically nodes will be more likely to be connected to each other if they are in the same community and less likely if in different communities.
    • The algorithm works by finding an edge between communities and then removes these edges leaving behind only the communities themselves. For this it uses is Betweenness.
    • Betweenness assigns a large number to edges if they are between many pair of nodes.
    • Popular but slow takes O(m²n) on a network of n vertices and m edges making it impractical for a large set of nodes.
    • It focuses on these edges that are least central, the edges that are most “between” communities. The communities are detected by progressively removing edges from the original graph
    • If a network contains communities or groups that are only loosely connected by a few intergroup edges, then all shortest paths between different communities must go along one of these few edges. Thus, the edges connecting communities will have high edge betweenness (at least one of them). By removing these edges, the groups are separated from one another and so the underlying community structure of the network is revealed.
    • 

      http://igraph.sourceforge.net/doc/R/betweenness.html
    • http://www.sixhat.net/finding-communities-in-networks-with-r-and-igraph.html
    • 

      http://arxiv.org/pdf/cond-mat/0309045.pdf
    • http://open.umich.edu/education/si/si508/fall2008/materials#Labs

Can the new NoSQL databases formats like key-store, graph based, column family datastore, and document oriented databases compete with already optimized relational databases like oracle, MySQL etc.

The traditional relational databases have little room for improvement as they are highly optimized and are already in place in number of applications but these relational databases don’t scale well that is where the different type of NoSQL databases come in they don’t some things like transactions or comprise on some features but they are built to be fast and scalable. Also, with the fast speed of changes in applications and need to adapt the database quickly to changing requirements and changing database structure the traditional databases are more difficult to change. Changing the traditional databases requires changing the whole database structure, change all the applications which uses the applications and this makes it harder for the databases to accomodate change whereas in less critical areas like in social network data where change is normal and using traditional databases is too difficult to use and to scale NoSQL provides the flexibility to be able to add change the structure for new data and merge different format of data without changing existing applications. Sharding and replication work well with large databases. With the progression of internet more and more data is collected by all organizations and existing databases fail to accomodate Big data. In order to accomodate big data there is a need to use technologies like NoSQL databases, hadoop, map-reduce and similar techniques to reduce the problems in smaller chunks and use cloud computing to do what is not possible to do anymore in traditional databases.

In the past if you had a lot of data with a lot of columns and based on the columns you wanted to find a pattern between the variables and the output we are interested to analyze you would use statistical analysis. These statistical models are too difficult to use when the data approaches a large scale i.e. Big data. Big data makes the statistical models slow to use and impossible to use. So in order to use them there is a need to make some kind of algorithms which distribute the data in buckets, uses hadoop and map-reduce to apply some kind of calculation we are interested in and apply them to smaller problems, finding the result and merging them to get the result we want. This involves now use of cloud computing.