diff --git a/BUILD.md b/BUILD.md new file mode 100755 index 0000000..abc6774 --- /dev/null +++ b/BUILD.md @@ -0,0 +1,117 @@ +# Notes for building snappy-java + +snappy-java supports Windows, Mac OS X, Linux (x86, x86_64, arm, etc...). If your platform is not supported, you need to build native libraries by yourself. + +## Requited Tools + - Java 7 or higher + - Maven3 (mvn) + - GNU make, autotools + + +## Building snappy-java + +To build jar file of snappy-java, type: +$ make + +A native library for your machine environment and a jar package target/snappy-java-(version).jar are produced in the target folder. + + +### Rebuild the native library for your platform +``` +$ make clean-native native +``` + +## Platform specific tips + +After snappy-java 1.1.3, we are using docker images of cross compilers. So no longer need to build native libraries by actually running the target OS. +The following notes are obsolete, but preserved here for future references. + +### Windows (32/64-bit) +* GNU make +* And also tar, curl, cp, rm, grep commands are needed. (I use Cygwin and MinGW for building snappy-java in Windows) + +### Windows (32-bit only) +* Install MinGW http://www.mingw.org/ +* Set PATH to the following command in MinGW package + - mingw32-g++ + - strip + +To build x86 (32bit) dll under 64-bit Windows, use "make win32" target. + +### Windows (64-bit only) +* Download MinGW-w64 http://sourceforge.net/projects/mingw-w64/ +* Set PATH to the following commands in the downloaded archive: + - x86_64-w64-mingw32-g++ + - x86_64-w64-mingw32-strip + +NOTICE: Do not use the Cygwin version of MinGW-w64. It fails to build assemblies for 64bit environment. + +### Linux (32/64-bit) +* gcc-4.5.x or higher is necessary because snappy-java uses -static-libstdc++ option. It is possible to use gcc-4.3.x but a dependency to libstdc++ remains in the generated jar file; That means if another version of libstdc++ is used, snappy-java might not work correctly. +* You can build 32-bit native library with 64-bit Linux machine (do make linux32) + +### Mac +* Install gcc, make, etc. included in Mac OS X install disk. (X Code). And also intall libtool: +``` +$ brew install libtool +``` + +## Building Linux x86\_64 binary + +(obsolete: snappy-java now uses a docker image `xerial/centos5-linux-x86_86-pic` which contains g++ built with `-fPIC` option. ) + +snappy-java tries to static link libstdc++ to increase the availability for various Linux versions. However, standard distributions of 64-bit Linux OS rarely provide libstdc++ compiled with `-fPIC` option. I currently uses custom g++, compiled as follows: + +``` +$ cd work +$ wget (gcc-4.8.3 source) +$ tar xvfz (gcc-4.8.3.tar.gz) +$ cd gcc-4.8.3 +$ ./contrib/download_prerequisites +$ cd .. +$ mkdir objdir +$ cd objdir +$ ../gcc-4.8.3/configure --prefix=$HOME/local/gcc-4.8.3 CXXFLAGS=-fPIC CFLAGS=-fPIC --enable-languages=c,c++ +$ make +$ make install +``` + +This g++ build enables static linking of libstdc++. For more infomation on building GCC, see GCC's home page. + +## Building Linux s390/s390x binaries + +Older snapshots of snappy contain a buggy config.h.in that does not work properly on some big-endian platforms like Linux on IBM z (s390/s390x). Building snappy-java on s390/s390x requires fetching the snappy source from GitHub, and processing the source with autoconf to obtain a usable config.h. On a RHEL s390x system, these steps produced a working 64-bit snappy-java build (the process should be similar for other distributions): + + $ sudo yum install java-1.7.1-ibm-devel libstdc++-static-devel + $ export JAVA_HOME=/usr/lib/jvm/java-1.7.1-ibm-1.7.1.2.10-1jpp.3.el7_0.s390x + $ make USE_GIT=1 GIT_REPO_URL=https://github.com/google/snappy.git GIT_SNAPPY_BRANCH=master IBM_JDK_7=1 + +## Activating SSE2/AVX2 instructions in BitShuffle + +The most of the native libraries that snappy-java contains disable SSE2/AVX2 instructions in terms of portability (SSE2 is enabled only in Linux/x86_64 platforms). To enable AVX2 instructions, you need to compile as follows: + + $ make CXXFLAGS_BITSHUFFLE=-mavx2 # -msse2 for SSE2 instructions + +## Cross-compiling for other platforms (obsolete) +The Makefile contains rules for cross-compiling the native library for other platforms so that the snappy-java JAR can support multiple platforms. For example, to build the native libraries for x86 Linux, x86 and x86-64 Windows, and soft- and hard-float ARM: + + $ make linux32 win32 win64 linux-arm linux-armhf linux-aarch64 + +If you append `snappy` to the line above, it will also build the native library for the current platform and then build the snappy-java JAR (containing all native libraries built so far). + +Of course, you must first have the necessary cross-compilers and development libraries installed for each target CPU and OS. For example, on Ubuntu 12.04 for x86-64, install the following packages for each target: + + * linux32: `sudo apt-get install g++-multilib libc6-dev-i386 lib32stdc++6` + * win32: `sudo apt-get install g++-mingw-w64-i686` + * win64: `sudo apt-get install g++-mingw-w64-x86-64` + * arm: `sudo apt-get install g++-arm-linux-gnueabi` + * armhf: `sudo apt-get install g++-arm-linux-gnueabihf` + * aarch64: `sudo apt-get install g++-aarch64-linux` + +Unfortunately, cross-compiling for Mac OS X is not currently possible; you must compile within OS X. + +If you are using Mac and openjdk7 (or higher), use the following option: + + $ make native LIBNAME=libsnappyjava.dylib + + diff --git a/INSTALL b/INSTALL deleted file mode 100755 index 21c6d8c..0000000 --- a/INSTALL +++ /dev/null @@ -1,78 +0,0 @@ -[Installation note of snappy-java] - -If your OS platform is one of the Windows/Linux/Mac OS X (32/64 bit Intel CPUs), the installation process described here is unnecessary. Simply put snappy-java-(version).jar to your Java classpath. If your OS is not listed above, create your own snappy-java-(version).jar file as follows. - - -= Required tools for building snappy-java = - -[For all platforms] - -* Java 6 (JDK1.6) http://java.sun.com/ - - set JAVA_HOME environment variable to the Java installation folder (e.g. JAVA_HOME=C:/Program Files/Java/jdk1.6.0_24 in Windows) -* Maven 3.x http://maven.apache.org/ - - Check mvn command can be used from your command line. - -[Windows (32/64-bit)] -* GNU make -* And also tar, curl, cp, rm, grep commands are needed. (I use Cygwin and MinGW for building snappy-java in Windows) - -[Windows (32-bit only)] -* Install MinGW http://www.mingw.org/ -* Set PATH to the following command in MinGW package - - mingw32-g++ - - strip - -To build x86 (32bit) dll under 64-bit Windows, use "make win32" target. - -[Windows (64-bit only)] -* Download MinGW-w64 http://sourceforge.net/projects/mingw-w64/ -* Set PATH to the following commands in the downloaded archive: - - x86_64-w64-mingw32-g++ - - x86_64-w64-mingw32-strip - -NOTICE: Do not use the Cygwin version of MinGW-w64. It fails to build assemblies for 64bit environment. - -[Linux (32/64-bit)] -* gcc-4.5.x or higher is necessary because snappy-java uses -static-libstdc++ option. It is possible to use gcc-4.3.x but a dependency to libstdc++ remains in the generated jar file; That means if another version of libstdc++ is used, snappy-java might not work correctly. -* You can build 32-bit native library with 64-bit Linux machine (do make linux32) - -[Mac] -* Install gcc, make, etc. included in Mac OS X install disk. (X Code) -* Install mercurial using Mac Ports http://www.macports.org/ - -= Building snappy-java = - -To build jar file of snappy-java, type: -$ make - -A native library for your machine environment and a jar package target/snappy-java-(version).jar are produced in the target folder. - -= Building only the native library = -$ make native - -= Rebuild the native library for your platform = -$ make clean-native native - - -= Using system installed libsnappyjava (or snappyjava.dll) = - -Set org.xerial.snappy.use.systemlib system property to true: - -java -Djava.library.path=(path to the installed snappyjava lib) -Dorg.xerial.snappy.use.systemlib=true ... - -With this setting snappy-java does not use bundled native libraries. Insted it tries to load native library installed at the path specified in java.library.path. - - -= Configure snappy-java using property file = - -Prepare org-xerial-snappy.properties file (under the root path of your library) in Java's property file format. -Here is a list of the available properties: - - * org.xerial.snappy.lib.path (directory containing a snappyjava's native library) - * org.xerial.snappy.lib.name (library file name) - * org.xerial.snappy.tempdir (temporary directory to extract a native library bundled in snappy-java) - * org.xerial.snappy.use.systemlib (if this value is true, use system installed libsnappyjava.so looking the path specified by java.library.path) - - - - diff --git a/Makefile b/Makefile index 61e18bd..3f27aca 100644 --- a/Makefile +++ b/Makefile @@ -146,7 +146,7 @@ $(TARGET)/$(snappy-jar-version).jar: test: $(NATIVE_DLL) $(SBT) test -DOCKER_RUN_OPTS=--rm +DOCKER_RUN_OPTS:=--rm win32: jni-header ./docker/dockcross-windows-x86 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=i686-w64-mingw32.static- OS_NAME=Windows OS_ARCH=x86' diff --git a/README.md b/README.md index 378739d..bc6b1d5 100755 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ -The snappy-java is a Java port of the snappy +snappy-java +=== +snappy-java is a Java port of the snappy , a fast C++ compresser/decompresser developed by Google. ## Features @@ -94,83 +96,11 @@ If you have snappy-java-(VERSION).jar in the current directory, use `-classpath` $ javac -classpath ".:snappy-java-(VERSION).jar" Sample.java # in Mac or Linux - - ## Public discussion group Post bug reports or feature request to the Issue Tracker: Public discussion forum is here: [Xerial Public Discussion Group](http://groups.google.com/group/xerial?hl=en) - -## Building from the source code -See the [installation instruction](https://github.com/xerial/snappy-java/blob/develop/INSTALL). Building from the source code is an option when your OS platform and CPU architecture is not supported. To build snappy-java, you need Git, JDK (1.6 or higher), g++ compiler (mingw in Windows) etc. - - $ git clone https://github.com/xerial/snappy-java.git - $ cd snappy-java - $ make - -When building on Solaris use - - $ gmake - -A file `target/snappy-java-$(version).jar` is the product additionally containing the native library built for your platform. - -## Building Linux x86\_64 binary - -snappy-java tries to static link libstdc++ to increase the availability for various Linux versions. However, standard distributions of 64-bit Linux OS rarely provide libstdc++ compiled with `-fPIC` option. I currently uses custom g++, compiled as follows: - -``` -$ cd work -$ wget (gcc-4.8.3 source) -$ tar xvfz (gcc-4.8.3.tar.gz) -$ cd gcc-4.8.3 -$ ./contrib/download_prerequisites -$ cd .. -$ mkdir objdir -$ cd objdir -$ ../gcc-4.8.3/configure --prefix=$HOME/local/gcc-4.8.3 CXXFLAGS=-fPIC CFLAGS=-fPIC --enable-languages=c,c++ -$ make -$ make install -``` - -This g++ build enables static linking of libstdc++. For more infomation on building GCC, see GCC's home page. - -## Building Linux s390/s390x binaries - -Older snapshots of snappy contain a buggy config.h.in that does not work properly on some big-endian platforms like Linux on IBM z (s390/s390x). Building snappy-java on s390/s390x requires fetching the snappy source from GitHub, and processing the source with autoconf to obtain a usable config.h. On a RHEL s390x system, these steps produced a working 64-bit snappy-java build (the process should be similar for other distributions): - - $ sudo yum install java-1.7.1-ibm-devel libstdc++-static-devel - $ export JAVA_HOME=/usr/lib/jvm/java-1.7.1-ibm-1.7.1.2.10-1jpp.3.el7_0.s390x - $ make USE_GIT=1 GIT_REPO_URL=https://github.com/google/snappy.git GIT_SNAPPY_BRANCH=master IBM_JDK_7=1 - -## Activating SSE2/AVX2 instructions in BitShuffle - -The most of the native libraries that snappy-java contains disable SSE2/AVX2 instructions in terms of portability (SSE2 is enabled only in Linux/x86_64 platforms). To enable AVX2 instructions, you need to compile as follows: - - $ make CXXFLAGS_BITSHUFFLE=-mavx2 # -msse2 for SSE2 instructions - -## Cross-compiling for other platforms -The Makefile contains rules for cross-compiling the native library for other platforms so that the snappy-java JAR can support multiple platforms. For example, to build the native libraries for x86 Linux, x86 and x86-64 Windows, and soft- and hard-float ARM: - - $ make linux32 win32 win64 linux-arm linux-armhf linux-aarch64 - -If you append `snappy` to the line above, it will also build the native library for the current platform and then build the snappy-java JAR (containing all native libraries built so far). - -Of course, you must first have the necessary cross-compilers and development libraries installed for each target CPU and OS. For example, on Ubuntu 12.04 for x86-64, install the following packages for each target: - - * linux32: `sudo apt-get install g++-multilib libc6-dev-i386 lib32stdc++6` - * win32: `sudo apt-get install g++-mingw-w64-i686` - * win64: `sudo apt-get install g++-mingw-w64-x86-64` - * arm: `sudo apt-get install g++-arm-linux-gnueabi` - * armhf: `sudo apt-get install g++-arm-linux-gnueabihf` - * aarch64: `sudo apt-get install g++-aarch64-linux` - -Unfortunately, cross-compiling for Mac OS X is not currently possible; you must compile within OS X. - -If you are using Mac and openjdk7 (or higher), use the following option: - - $ make native LIBNAME=libsnappyjava.dylib - ## For developers snappy-java uses sbt (simple build tool for Scala) as a build tool. Here is a simple usage @@ -191,6 +121,19 @@ $ ./sbt -Dloglevel=debug For the details of sbt usage, see my blog post: [Building Java Projects with sbt](http://xerial.org/blog/2014/03/24/sbt/) +### Building from the source code +See the [build instruction](https://github.com/xerial/snappy-java/blob/master/BUILD.md). Building from the source code is an option when your OS platform and CPU architecture is not supported. To build snappy-java, you need Git, JDK (1.6 or higher), g++ compiler (mingw in Windows) etc. + + $ git clone https://github.com/xerial/snappy-java.git + $ cd snappy-java + $ make + +When building on Solaris, use `gmake`: + + $ gmake + +A file `target/snappy-java-$(version).jar` is the product additionally containing the native library built for your platform. + ## Miscellaneous Notes ### Using snappy-java with Tomcat 6 (or higher) Web Server @@ -198,3 +141,15 @@ Simply put the snappy-java's jar to WEB-INF/lib folder of your web application. ---- Snappy-java is developed by [Taro L. Saito](http://www.xerial.org/leo). Twitter [@taroleo](http://twitter.com/#!/taroleo) + + +### Configure snappy-java using property file + +Prepare org-xerial-snappy.properties file (under the root path of your library) in Java's property file format. +Here is a list of the available properties: + + * org.xerial.snappy.lib.path (directory containing a snappyjava's native library) + * org.xerial.snappy.lib.name (library file name) + * org.xerial.snappy.tempdir (temporary directory to extract a native library bundled in snappy-java) + * org.xerial.snappy.use.systemlib (if this value is true, use system installed libsnappyjava.so looking the path specified by java.library.path) +