searchcode server

What is it?

searchcode server is a powerful code search engine with a sleak web user interface.

searchcode server works in tandem with your source control system, indexing thousands of repositories and files allowing you and your developers to quickly find and reuse code across teams.

See https://searchcode.com/product/ for more details

Support

If you are building from source then you get no support and must work within the restrictions specified of the fair source licence (see LICENSE.txt for details). To purchase support see https://searchcode.com/product/#downloadlinks

Submissions/Pull Requests

If you would like to submit code to be integrated into searchcode server please specify that it is dual licenced under Apache Public License v2 and GPL v3. This allows it to be pulled in directly without having to worry about licencing issues in the future.

Before making any submission be sure to run "fab test"

To Build TL/DR;

Install Java 8 (Oracle or OpenJDK)
Install Maven
Install Python and Fabric (optional)
Install NPM and run "npm install -g node-qunit-phantomjs" (optional)
mvn test OR fab test (if fabric) to run unit tests
fab test_full (if fabric) to run all tests (need to be running application in background first)

To Build

To build searchcode server you need to have any Windows, Linux or OSX machine with Java 8 installed and maven setup. Idealy you want Python and Python Fabric installed as well but it is not a requirement. To run the javascript unit tests through the command line you will need to install NPM and then run npm install -g node-qunit-phantomjs but you can always run them in your browser if required by opening ./src/test/javascript/index.html

To test the application you can either run "mvn test" or "fab test". Note that these tests will only cover unit some integration and javascript unit. For full coverage run "fab test_full" with the application running in the background to ensure everything is working as expected.

To build a full release IE ready for production you should run "fab build_release" which will test compile and build a release into the release folder and produce the file "searchcode-server.tar.gz" which is a ready to deploy release.

If you want to simply test and run then you can run "fab run" however this will be default build a package and run that. To run quickly just open in your IDE of choice and start running App.java

There are a special bunch of tests used for verifying that indexing logic for GIT and File repositories works correctly. To perform this you need to run one of all of the following shell scripts,

./assets/integration_test/gitload/gitload.sh ./assets/integration_test/gitupdate/gitupdate.sh ./assets/integration_test/fileupdatetest/fileload.sh ./assets/integration_test/fileload/fileload.sh

Then add the git ones as GIT repositories in the application and the file ones as FILE repositories. It is also useful to set the properties

check_repo_chages=60 check_filerepo_chages=60

but it is not required. Then run searchcode. The scripts will add/remove/update files every 60 seconds which should force searchcode to add/update/remove files from the index in an attempt to hit as many code paths as possible. With this done there should be no more than 400 documents indexed at any time (if indexing all 4 repositories) and a minimum of 201 (the fileload.sh files + fileupdatetest files + gitupdate files). Leave things running over several hours to ensure that the logic works correctly.

Before a release is made a build must pass all of the above checks, with the indexer logic tests being run for a minimum of 24 hours. To ensure that performance is acceptable the tests are also run on a Atom powered netbook.

To Run

To run searchcode server you will need to have any Windows, Linux or OSX machine with Java 8 installed. Testing and packaging was done using the below version.

java version "1.8.0_65" Java(TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

Uncompress the file you have downloaded to a directory where you want to run searchcode server. This directory should have more disk space than the size of the repositories you want to index.

Once unpacked assuming that java is in your path (check with the command java -version) you should be able to run searchcode with the following command for Linux/OSX/BSD

./searchcode-server.sh

or for Windows

searchcode-server.bat

After a few moments searchcode server should be ready to run. By default it will be running on port 8080. To connect to it enter the following in the browser,

http://SERVER_IP:8080

Be sure to replace SERVER_IP with the ip address if your server or localhost if running locally. If you see a page with a search bar then everything is fine.

For further control you may want to edit the above files and include the java -Xmx argument to specify the amount of RAM to use or any other java option you wish to pass in.

To administer your searchcode server instance you need to click on the Admin link in the top right. Enter the default password Adm1n234 (change this via the properties file) to add git repositories. If you need help check the documentation page (link at the bottom of every page).

Upgrades

Always keep in mind that upgrades are best done with a full reindex. You can either do this by deleting the contents of the index directory that you have configured in your searchcode.properties file or by clicking the "Recrawl & Rebuild Indexes" button in the admin screen.

To upgrade your current instance of searchcode perform the following steps.

Stop your current instance of searchcode server
Make a backup copy of your current instances searchcode.properties and searchcode.sqlite files.
Uncompress the package to a new directory.
You can either
- Copy the uncompressed files over your current instance overwriting if prompted.
- Copy the directory dependancy-jars and all contents overwriting your current
- Copy the following files searchcode-1.3.6.jar searchcode-server.bar and searchcode-server.sh to your instance directory
Start your instance again
Login to the admin screen and click the "Recrawl & Rebuild Indexes" button

It is also worth comparing your searchcode.properties file to the new one (or the documentation page) as there may be new configuration that you can use. Also be sure to check the settings page as there is likely to be new settings you can use.

Licence

Use of this software is governed by the Fair Source License included in the LICENSE.txt file

In order to deal with the case of my death or this software becoming abandoned it has an open eventually clause where the licence will change exactly 3 years after the publish date of a version release. This means that if version 1.0.0 was released on 1 July 2010 then it can be taken using the listed alternate licence on 2 July 2013. This licence, version and time is all specified below.

After the following date DATE NOT SPECIFIED YET this software version '1.3.6' is dual licenced under the Fair Source Licence included in the LICENSE.txt file or under the GNU General Public License Version 3 with terms specified at https://www.gnu.org/licenses/gpl-3.0.txt

OWASP Database is licensed under https://creativecommons.org/licenses/by-sa/3.0/ sourced under Creative Commons from https://codecrawler.codeplex.com/ https://www.owasp.org/index.php/Category:OWASP_Code_Crawler and https://www.owasp.org/index.php/OWASP_Code_Review_Guide_Table_of_Contents this database was modified to JSON and with slight corrections to spelling and puncuation where applicable.

Change Log

XX XXXXXXXX XXXX - 1.3.6

Add job to scan repositories updating the spelling corrector
Add logic to split interesting characters such as Chinese/Korean/Japanese to allow searching
Fix XSS on admin log page
Fix XSS on admin repo page
Fix bug where unable to delete repositories on admin page with special characters in name
Running jobs now displayed on admin page
Repository overview page added to get overview of a repository

19 December 2016 - 1.3.5

Fix only_localhost to really only listen localhost not drop connections
Add SHA512 hmac as option for signed API requests
Resolve XSS vulnerability issue
Add logic to deal with immutable files by moving to trash folder
Add log to display most recent search results
Resolve several memory leak issues
Modify logic to allow matching URL_CONFIG in App.URL_CONFIG.toString()
Modify logic to allow matching install_something in :install_something
Logging level of OFF now does not even attempt to create log file
Resolve uncaught exception when writing log
Close resource/memory leak in opened files
Changed removal logic for removed files to be less aggressive
Fix potential overflow for search query count (however unlikely that is)
Fix race conditions with logging

21 November 2016 - 1.3.4

Refactor index logic to resolve removal bugs
Optimise memory usage of file repositories
Fix headers for API responses
Change admin fields from disabled to readonly to allow copy paste
Fix bug where sqlite_file property was being ignored

02 November 2016 - 1.3.3

Add log level selection to admin logs page
Can now search across filenames and paths
Slight performance improvements for indexing
Better error reporting for repository errors
Better guessing of file encoding type
White/Black list of file extensions
Fix bug where OWASP results not escaped
Add logout button to admin sections
Refactor JavaScript and fix bugs
Additional properties for controlling background threads
Fix off by 1 error where some results would not be shown
Admin can now selectively pause indexing
Ability to log out all indexed and excluded files to CSV
Set file rotation to have 10x10mb files by default but controlled through property
Add backoff setting to settings page to control CPU usage
Improve memory usage for large file repositories
Whitelist trumps minified logic

04 October 2016 - 1.3.2

Improve matching of certain strings such as E.G. linux2.7.4 or i386
Fix bug with binary file checking

27 September 2016 - 1.3.1

Fix potential performance issue in spelling corrector with long strings
Fix potential XSS issue with HTML page
Line numbers now appear on code result pages
Logging directory not configurable via properties
Symlinks path traversal issue resolved
Highlight.js now lazy loads
Highlight limit now configrable through properties
OWASP line number display correction
Header links resolved in some cases

20 September 2016 - 1.3.0

Filters now apply instantly when clicked option is toggleable
OWASP checking on code result pages based on OWASP code crawler
Log Tail screen added to Admin
Add server date time output to admin page
Add rebuild all button to Admin page
Fix bug where clicking deep links on HTML page would break
Modify file links to use unique codeid to avoid click through issues
Improve indexing performance by intelligently calculating MD5
Add ability to index arbitary file system locations
Add API endpoint to force reindex

30 August 2016 - 1.2.4

Add gzip compression to most used routes
Refactor repository jobs to resolve bugs and make more extensible
Fix lines to index bug with SVN which could result in index pausing
Update HTML page to have new look and feel

23 August 2016 - 1.2.3

Upgrade mithril to 0.2.5 to resolve Safari bug where pages would not render
Release under fair source license
Set search box to use monospaced font

17 August 2016 - 1.2.2

Add -w option to external git to avoid whitespace blame issue
Resolve bug where some results were searchable but not browsable
Resolve SVN cert prompt issue causing SVN to never checkout
Fix potential performance issues with spell checker

28 June 2016 - 1.2.1

Resolve string exception bug when using local git on some files
Resolve bug where empty username, language or repository caused indexing to throw exception
Add additional information about resolving ulimit errors to documentation

09 June 2016 - 1.2.0

Add SVN repository support
Resolve highlight issue where overlapping matching text caused issues
Add search operators AND OR NOT and wildcard E.G. searc*
Resolve bug where some files when clicked would be labeled as 404 while being searchable

30 May 2016 - 1.1.3

New property added to control depth of the file to index

27 May 2016 - 1.1.2

Additional resolve bug on RedHat/CentOS where file owner always resolves as "unknown"
Additional memory improvements
New properties added to control size of indexing queue size

20 May 2016 - 1.1.1

Resolve bug on RedHat/CentOS where file owner always resolves as "unknown"
Adjust client side cache times to mitigate sleeping laptop cache issue
Small performance tweaks

13 May 2016 - 1.1.0

Updated to allow indexing of named branches
Additional memory improvements
Logging of exceptions/warnings to rolling file
Logging level support added via properties file
Caching improvements
Resolve issue where sometimes .git directory files would be indexed
Improve delta calculation between revisions to improve indexing speed
Fix bug where deleted repository indexing jobs would continue to run
Fix bug where missing source could cause clicking through to result bug
Additional file types added to file classifier
Improved COCOMO calculation
API Endpoint addition allowing adding, removing and list of repositories
Increase indexing priority to ensure index refreshes happen faster
Allow use of tilde ~ in repository names
Improvements to index/clone when server unexpectedly stopped
Paging added to the repository page
Many additional bug fixes

20 April 2016 - 1.0.3

Improve memory performance across indexing and searching
Add additional parameters allowing memory usage to be configured
Resolve null pointer exceptions in various areas of the search pipeline
Fix removal of repository logic

14 April 2016 - 1.0.2

Fix bug where unable to add new repository

13 April 2016 - 1.0.1

Updated indexer code to improve performance
Fix MD5 file calculation bug
Add owner and user filter into index
Add Match Lines, Max Line Depth, Minified Length to settings page
Change links to files away from document id to path
Added splitting of terms where possible such as ArrayList into Array List
Improved match relevance for searches
Improved calculation of COCOMO to be more accurate
Fixed overloaded issues when server under high load
Updated language identifiation to pick up languages, Varnish Configuration, Razor Template, Jade Template, ReStructuredText, Razor Template, Handlebars Template
Performance improvements for all pages using smart caching of reused assets as found through load tests

31 March 2016 - 1.0.0

Updated look and feel with new fancy single page application design
Improved performance of search and match 3-20x
Additional configuration of properties file allowing reverting to pure HTML page if required
Various bug fixes

08 March 2016 - 0.9.9

Inital release

Join the chat at https://gitter.im/searchcode-server/Lobby

devheroo / searchcode-server