tyler.durden
/
snappy-java
mirror of https://github.com/xerial/snappy-java.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #151 from xerial/cross-build 

Cross building snappy-java using docker
This commit is contained in:
Taro L. Saito 2017-01-19 23:33:20 -08:00 committed by GitHub
parent f72229f913 e329283744
commit 8427461ece
36 changed files with 1991 additions and 522 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
.travis.yml
View File

@ -1,14 +1,25 @@
sudo: required
services:
- docker
language: scala
sudo: false
scala:
- 2.11.7
- 2.11.8
jdk:
- openjdk6
- openjdk7
- oraclejdk7
- oraclejdk8
script: ./sbt test
script:
- if [ "$TRAVIS_JDK_VERSION" == "oraclejdk8" ]; then
if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then
sbt ++$TRAVIS_SCALA_VERSION "; test; publish";
else
sbt ++$TRAVIS_SCALA_VERSION test;
fi;
else
sbt ++$TRAVIS_SCALA_VERSION test;
fi

119
BUILD.md Executable file
View File

@ -0,0 +1,119 @@
# Notes for building snappy-java
snappy-java supports Windows, Mac OS X, Linux (x86, x86_64, arm, etc...). If your platform is not supported, you need to build native libraries by yourself.
## Requited Tools
- Java 7 or higher
- Maven3 (mvn)
- GNU make, autotools
## Building snappy-java
To build jar file of snappy-java, type:
```
$ make
```
A native library for your machine environment and a jar package target/snappy-java-(version).jar are produced in the target folder.
### Rebuild the native library for your platform
```
$ make clean-native native
```
## Platform specific tips
After snappy-java 1.1.3, we are using docker images of cross compilers. So no longer need to build native libraries by actually running the target OS.
The following notes are obsolete, but preserved here for future references.
### Windows (32/64-bit)
* GNU make
* And also tar, curl, cp, rm, grep commands are needed. (I use Cygwin and MinGW for building snappy-java in Windows)
### Windows (32-bit only)
* Install MinGW http://www.mingw.org/
* Set PATH to the following command in MinGW package
- mingw32-g++
- strip
To build x86 (32bit) dll under 64-bit Windows, use "make win32" target.
### Windows (64-bit only)
* Download MinGW-w64 http://sourceforge.net/projects/mingw-w64/
* Set PATH to the following commands in the downloaded archive:
- x86_64-w64-mingw32-g++
- x86_64-w64-mingw32-strip
NOTICE: Do not use the Cygwin version of MinGW-w64. It fails to build assemblies for 64bit environment.
### Linux (32/64-bit)
* gcc-4.5.x or higher is necessary because snappy-java uses -static-libstdc++ option. It is possible to use gcc-4.3.x but a dependency to libstdc++ remains in the generated jar file; That means if another version of libstdc++ is used, snappy-java might not work correctly.
* You can build 32-bit native library with 64-bit Linux machine (do make linux32)
### Mac
* Install gcc, make, etc. included in Mac OS X install disk. (X Code). And also intall libtool:
```
$ brew install libtool
```
## Building Linux x86\_64 binary
(obsolete: snappy-java now uses a docker image `xerial/centos5-linux-x86_86-pic` which contains g++ built with `-fPIC` option. )
snappy-java tries to static link libstdc++ to increase the availability for various Linux versions. However, standard distributions of 64-bit Linux OS rarely provide libstdc++ compiled with `-fPIC` option. I currently uses custom g++, compiled as follows:
```
$ cd work
$ wget (gcc-4.8.3 source)
$ tar xvfz (gcc-4.8.3.tar.gz)
$ cd gcc-4.8.3
$ ./contrib/download_prerequisites
$ cd ..
$ mkdir objdir
$ cd objdir
$ ../gcc-4.8.3/configure --prefix=$HOME/local/gcc-4.8.3 CXXFLAGS=-fPIC CFLAGS=-fPIC --enable-languages=c,c++
$ make
$ make install
```
This g++ build enables static linking of libstdc++. For more infomation on building GCC, see GCC's home page.
## Building Linux s390/s390x binaries
Older snapshots of snappy contain a buggy config.h.in that does not work properly on some big-endian platforms like Linux on IBM z (s390/s390x). Building snappy-java on s390/s390x requires fetching the snappy source from GitHub, and processing the source with autoconf to obtain a usable config.h. On a RHEL s390x system, these steps produced a working 64-bit snappy-java build (the process should be similar for other distributions):
$ sudo yum install java-1.7.1-ibm-devel libstdc++-static-devel
$ export JAVA_HOME=/usr/lib/jvm/java-1.7.1-ibm-1.7.1.2.10-1jpp.3.el7_0.s390x
$ make USE_GIT=1 GIT_REPO_URL=https://github.com/google/snappy.git GIT_SNAPPY_BRANCH=master IBM_JDK_7=1
## Activating SSE2/AVX2 instructions in BitShuffle
The most of the native libraries that snappy-java contains disable SSE2/AVX2 instructions in terms of portability (SSE2 is enabled only in Linux/x86_64 platforms). To enable AVX2 instructions, you need to compile as follows:
$ make CXXFLAGS_BITSHUFFLE=-mavx2 # -msse2 for SSE2 instructions
## Cross-compiling for other platforms (obsolete)
The Makefile contains rules for cross-compiling the native library for other platforms so that the snappy-java JAR can support multiple platforms. For example, to build the native libraries for x86 Linux, x86 and x86-64 Windows, and soft- and hard-float ARM:
$ make linux32 win32 win64 linux-arm linux-armhf linux-aarch64
If you append `snappy` to the line above, it will also build the native library for the current platform and then build the snappy-java JAR (containing all native libraries built so far).
Of course, you must first have the necessary cross-compilers and development libraries installed for each target CPU and OS. For example, on Ubuntu 12.04 for x86-64, install the following packages for each target:
* linux32: `sudo apt-get install g++-multilib libc6-dev-i386 lib32stdc++6`
* win32: `sudo apt-get install g++-mingw-w64-i686`
* win64: `sudo apt-get install g++-mingw-w64-x86-64`
* arm: `sudo apt-get install g++-arm-linux-gnueabi`
* armhf: `sudo apt-get install g++-arm-linux-gnueabihf`
* aarch64: `sudo apt-get install g++-aarch64-linux`
Unfortunately, cross-compiling for Mac OS X is not currently possible; you must compile within OS X.
If you are using Mac and openjdk7 (or higher), use the following option:
$ make native LIBNAME=libsnappyjava.dylib

78
INSTALL
View File

@ -1,78 +0,0 @@
[Installation note of snappy-java]
If your OS platform is one of the Windows/Linux/Mac OS X (32/64 bit Intel CPUs), the installation process described here is unnecessary. Simply put snappy-java-(version).jar to your Java classpath. If your OS is not listed above, create your own snappy-java-(version).jar file as follows.
= Required tools for building snappy-java =
[For all platforms]
* Java 6 (JDK1.6) http://java.sun.com/
- set JAVA_HOME environment variable to the Java installation folder (e.g. JAVA_HOME=C:/Program Files/Java/jdk1.6.0_24 in Windows)
* Maven 3.x http://maven.apache.org/
- Check mvn command can be used from your command line.
[Windows (32/64-bit)]
* GNU make
* And also tar, curl, cp, rm, grep commands are needed. (I use Cygwin and MinGW for building snappy-java in Windows)
[Windows (32-bit only)]
* Install MinGW http://www.mingw.org/
* Set PATH to the following command in MinGW package
- mingw32-g++
- strip
To build x86 (32bit) dll under 64-bit Windows, use "make win32" target.
[Windows (64-bit only)]
* Download MinGW-w64 http://sourceforge.net/projects/mingw-w64/
* Set PATH to the following commands in the downloaded archive:
- x86_64-w64-mingw32-g++
- x86_64-w64-mingw32-strip
NOTICE: Do not use the Cygwin version of MinGW-w64. It fails to build assemblies for 64bit environment.
[Linux (32/64-bit)]
* gcc-4.5.x or higher is necessary because snappy-java uses -static-libstdc++ option. It is possible to use gcc-4.3.x but a dependency to libstdc++ remains in the generated jar file; That means if another version of libstdc++ is used, snappy-java might not work correctly.
* You can build 32-bit native library with 64-bit Linux machine (do make linux32)
[Mac]
* Install gcc, make, etc. included in Mac OS X install disk. (X Code)
* Install mercurial using Mac Ports http://www.macports.org/
= Building snappy-java =
To build jar file of snappy-java, type:
$ make
A native library for your machine environment and a jar package target/snappy-java-(version).jar are produced in the target folder.
= Building only the native library =
$ make native
= Rebuild the native library for your platform =
$ make clean-native native
= Using system installed libsnappyjava (or snappyjava.dll) =
Set org.xerial.snappy.use.systemlib system property to true:
java -Djava.library.path=(path to the installed snappyjava lib) -Dorg.xerial.snappy.use.systemlib=true ...
With this setting snappy-java does not use bundled native libraries. Insted it tries to load native library installed at the path specified in java.library.path.
= Configure snappy-java using property file =
Prepare org-xerial-snappy.properties file (under the root path of your library) in Java's property file format.
Here is a list of the available properties:
* org.xerial.snappy.lib.path (directory containing a snappyjava's native library)
* org.xerial.snappy.lib.name (library file name)
* org.xerial.snappy.tempdir (temporary directory to extract a native library bundled in snappy-java)
* org.xerial.snappy.use.systemlib (if this value is true, use system installed libsnappyjava.so looking the path specified by java.library.path)

104
Makefile
View File

@ -6,15 +6,16 @@ SBT:=./sbt
all: snappy
SNAPPY_OUT:=$(TARGET)/$(snappy)-$(os_arch)
SNAPPY_OUT:=$(TARGET)/snappy-$(SNAPPY_VERSION)-$(os_arch)
SNAPPY_ARCHIVE:=$(TARGET)/snappy-$(SNAPPY_VERSION).tar.gz
SNAPPY_CC:=snappy-sinksource.cc snappy-stubs-internal.cc snappy.cc
SNAPPY_SRC_DIR:=$(TARGET)/snappy-$(SNAPPY_VERSION)
SNAPPY_SRC:=$(addprefix $(SNAPPY_SRC_DIR)/,$(SNAPPY_CC))
SNAPPY_GIT_REPO_URL:=https://github.com/google/snappy
SNAPPY_GIT_REV:=2b9152d9c5bed71dffb7f7f6c7a3ec48b058ff2d # 1.1.3 with autogen.sh fix
SNAPPY_GIT_REV:=32d6d7d8a2ef328a2ee1dd40f072e21f4983ebda # 1.1.3 May 23, 2016
SNAPPY_UNPACKED:=$(TARGET)/snappy-extracted.log
SNAPPY_GIT_UNPACKED:=$(TARGET)/snappy-git-extracted.log
SNAPPY_SOURCE_CONFIGURED:=$(TARGET)/snappy-configure.log
BITSHUFFLE_ARCHIVE:=$(TARGET)/bitshuffle-$(BITSHUFFLE_VERSION).tar.gz
BITSHUFFLE_C:=bitshuffle_core.c iochain.c
@ -71,27 +72,29 @@ $(SNAPPY_GIT_UNPACKED):
rm -rf $(SNAPPY_SRC_DIR)
@mkdir -p $(SNAPPY_SRC_DIR)
git clone $(SNAPPY_GIT_REPO_URL) $(SNAPPY_SRC_DIR)
git --git-dir=$(SNAPPY_SRC_DIR)/.git --work-tree=$(SNAPPY_SRC_DIR) checkout -b local/snappy-$(VERSION) $(SNAPPY_GIT_REV)
git --git-dir=$(SNAPPY_SRC_DIR)/.git --work-tree=$(SNAPPY_SRC_DIR) checkout -b local/snappy-$(SNAPPY_VERSION) $(SNAPPY_GIT_REV)
touch $@
$(SNAPPY_SOURCE_CONFIGURED): $(SNAPPY_GIT_UNPACKED)
cd $(SNAPPY_SRC_DIR) && ./autogen.sh && ./configure
touch $@
jni-header: $(SRC)/org/xerial/snappy/SnappyNative.h $(SRC)/org/xerial/snappy/BitShuffleNative.h
jni-header: $(SNAPPY_SOURCE_CONFIGURED) $(BITSHUFFLE_UNPACKED) $(SRC)/org/xerial/snappy/SnappyNative.h $(SRC)/org/xerial/snappy/BitShuffleNative.h
$(TARGET)/jni-classes/org/xerial/snappy/SnappyNative.class: $(SRC)/org/xerial/snappy/SnappyNative.java
@mkdir -p $(TARGET)/jni-classes
$(JAVAC) -source 1.6 -target 1.6 -d $(TARGET)/jni-classes -sourcepath $(SRC) $<
$(JAVAC) -source 1.7 -target 1.7 -d $(TARGET)/jni-classes -sourcepath $(SRC) $<
$(SRC)/org/xerial/snappy/SnappyNative.h: $(TARGET)/jni-classes/org/xerial/snappy/SnappyNative.class
$(JAVAH) -force -classpath $(TARGET)/jni-classes -o $@ org.xerial.snappy.SnappyNative
$(TARGET)/jni-classes/org/xerial/snappy/BitShuffleNative.class: $(SRC)/org/xerial/snappy/BitShuffleNative.java
@mkdir -p $(TARGET)/jni-classes
$(JAVAC) -source 1.6 -target 1.6 -d $(TARGET)/jni-classes -sourcepath $(SRC) $<
$(JAVAC) -source 1.7 -target 1.7 -d $(TARGET)/jni-classes -sourcepath $(SRC) $<
$(SRC)/org/xerial/snappy/BitShuffleNative.h: $(TARGET)/jni-classes/org/xerial/snappy/BitShuffleNative.class
$(JAVAH) -force -classpath $(TARGET)/jni-classes -o $@ org.xerial.snappy.BitShuffleNative
$(SNAPPY_SRC): $(SNAPPY_GIT_UNPACKED)
$(SNAPPY_OUT)/%.o: $(SNAPPY_SRC_DIR)/%.cc
@mkdir -p $(@D)
@ -107,7 +110,10 @@ $(SNAPPY_OUT)/BitShuffleNative.o: $(SRC)/org/xerial/snappy/BitShuffleNative.cpp
$(SNAPPY_OUT)/$(LIBNAME): $(SNAPPY_OBJ)
$(CXX) $(CXXFLAGS) -o $@ $+ $(LINKFLAGS)
$(STRIP) $@
# Workaround for strip Protocol error when using VirtualBox on Mac
cp $@ /tmp/$(@F)
$(STRIP) /tmp/$(@F)
cp /tmp/$(@F) $@
clean-native:
rm -rf $(SNAPPY_OUT)
@ -121,15 +127,16 @@ NATIVE_DLL:=$(NATIVE_DIR)/$(LIBNAME)
snappy-jar-version:=snappy-java-$(shell perl -npe "s/version in ThisBuild\s+:=\s+\"(.*)\"/\1/" version.sbt | sed -e "/^$$/d")
native: $(SNAPPY_GIT_UNPACKED) $(NATIVE_DLL)
native: jni-header $(NATIVE_DLL)
snappy: native $(TARGET)/$(snappy-jar-version).jar
$(NATIVE_DLL): $(SNAPPY_OUT)/$(LIBNAME)
@mkdir -p $(@D)
cp $< $@
@mkdir -p $(NATIVE_TARGET_DIR)
cp $< $(NATIVE_TARGET_DIR)/$(LIBNAME)
native-all: win32 win64 mac64 native-arm linux32 linux64 linux-ppc64 linux-aarch64
$(NATIVE_DLL): $(SNAPPY_SOURCE_CONFIGURED) $(SNAPPY_OUT)/$(LIBNAME)
@mkdir -p $(@D)
cp $(SNAPPY_OUT)/$(LIBNAME) $@
@mkdir -p $(NATIVE_TARGET_DIR)
cp $(SNAPPY_OUT)/$(LIBNAME) $(NATIVE_TARGET_DIR)/$(LIBNAME)
package: $(TARGET)/$(snappy-jar-version).jar
@ -139,58 +146,53 @@ $(TARGET)/$(snappy-jar-version).jar:
test: $(NATIVE_DLL)
$(SBT) test
win32:
$(MAKE) native CROSS_PREFIX=i686-w64-mingw32- OS_NAME=Windows OS_ARCH=x86
DOCKER_RUN_OPTS:=--rm
# for cross-compilation on Ubuntu, install the g++-mingw-w64-x86-64 package
win64:
$(MAKE) native CROSS_PREFIX=x86_64-w64-mingw32- OS_NAME=Windows OS_ARCH=x86_64
win32: jni-header
./docker/dockcross-windows-x86 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=i686-w64-mingw32.static- OS_NAME=Windows OS_ARCH=x86'
mac32:
win64: jni-header
./docker/dockcross-windows-x64 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=x86_64-w64-mingw32.static- OS_NAME=Windows OS_ARCH=x86_64'
# deprecated
mac32: jni-header
$(MAKE) native OS_NAME=Mac OS_ARCH=x86
linux32:
$(MAKE) native OS_NAME=Linux OS_ARCH=x86
mac64: jni-header
docker run -it $(DOCKER_RUN_OPTS) -v $$PWD:/workdir -e CROSS_TRIPLE=x86_64-apple-darwin multiarch/crossbuild make clean-native native OS_NAME=Mac OS_ARCH=x86_64
linux32: jni-header
docker run $(DOCKER_RUN_OPTS) -ti -v $$PWD:/work xerial/centos5-linux-x86_64-pic bash -c 'make clean-native native OS_NAME=Linux OS_ARCH=x86'
linux64: jni-header
docker run $(DOCKER_RUN_OPTS) -ti -v $$PWD:/work xerial/centos5-linux-x86_64-pic bash -c 'make clean-native native OS_NAME=Linux OS_ARCH=x86_64'
freebsd64:
$(MAKE) native OS_NAME=FreeBSD OS_ARCH=x86_64
# for cross-compilation on Ubuntu, install the g++-arm-linux-gnueabi package
linux-arm:
$(MAKE) native CROSS_PREFIX=arm-linux-gnueabi- OS_NAME=Linux OS_ARCH=arm
# For ARM
native-arm: linux-arm linux-armv6 linux-armv7 linux-android-arm
# for cross-compilation on Ubuntu, install the g++-arm-linux-gnueabihf package
linux-armhf:
$(MAKE) native CROSS_PREFIX=arm-linux-gnueabihf- OS_NAME=Linux OS_ARCH=armhf
linux-arm: jni-header
./docker/dockcross-armv5 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=arm-linux-gnueabi- OS_NAME=Linux OS_ARCH=arm'
# for cross-compilation on Ubuntu, install the g++-aarch64-linux-gnu
linux-aarch64:
$(MAKE) native CROSS_PREFIX=aarch64-linux-gnu- OS_NAME=Linux OS_ARCH=aarch64
linux-armv6: jni-header
./docker/dockcross-armv6 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=arm-linux-gnueabihf- OS_NAME=Linux OS_ARCH=armv6'
clean-native-linux32:
$(MAKE) clean-native OS_NAME=Linux OS_ARCH=x86
linux-armv7: jni-header
./docker/dockcross-armv7 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=arm-linux-gnueabihf- OS_NAME=Linux OS_ARCH=armv7'
clean-native-win32:
$(MAKE) clean-native OS_NAME=Windows OS_ARCH=x86
linux-android-arm: jni-header
./docker/dockcross-android-arm -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=/usr/arm-linux-androideabi/bin/arm-linux-androideabi- OS_NAME=Linux OS_ARCH=android-arm'
linux-ppc64: jni-header
./docker/dockcross-ppc64 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=powerpc64le-linux-gnu- OS_NAME=Linux OS_ARCH=ppc64'
linux-aarch64: jni-header
./docker/dockcross-aarch64 -a $(DOCKER_RUN_OPTS) bash -c 'make clean-native native CROSS_PREFIX=aarch64-linux-gnu- OS_NAME=Linux OS_ARCH=aarch64'
javadoc:
$(SBT) doc
install-m2:
$(SBT) publishM2
googlecode-upload: googlecode-lib-upload googlecode-src-upload
googlecode-lib-upload: $(TARGET)/snappy-java-$(SNAPPY_VERSION)-lib.upload
googlecode-src-upload: $(TARGET)/snappy-java-$(SNAPPY_VERSION)-src.upload
GOOGLECODE_USER:=leo@xerial.org
$(TARGET)/snappy-java-$(SNAPPY_VERSION)-lib.upload:
./googlecode_upload.py -s "library for all platforms" -p snappy-java -l "Type-Executable,Featured,OpSys-All" -u "$(GOOGLECODE_USER)" target/snappy-java-$(SNAPPY_VERSION).jar
touch $@
$(TARGET)/snappy-java-$(SNAPPY_VERSION)-src.upload:
./googlecode_upload.py -s "source code archive" -p snappy-java -l "Type-Source,OpSys-All" -u "$(GOOGLECODE_USER)" target/snappy-java-$(SNAPPY_VERSION).tar.gz
touch $@

42
Makefile.common
View File

@ -3,7 +3,7 @@ SRC:=src/main/java
include src/main/resources/org/xerial/snappy/VERSION
ifndef JAVA_HOME
$(error Set JAVA_HOME environment variable)
$(warning Set JAVA_HOME environment variable)
endif
@ -42,7 +42,7 @@ endif
# os=Default is meant to be generic unix/linux
known_os_archs := Linux-x86 Linux-x86_64 Linux-arm Linux-armhf Linux-aarch64 Linux-ppc Linux-ppc64 Linux-s390 Linux-s390x Mac-x86 Mac-x86_64 FreeBSD-x86_64 Windows-x86 Windows-x86_64 SunOS-x86 SunOS-sparc SunOS-x86_64 AIX-ppc AIX-ppc64
known_os_archs := Linux-x86 Linux-x86_64 Linux-arm Linux-armv6 Linux-armv7 Linux-android-arm Linux-aarch64 Linux-ppc Linux-ppc64 Linux-s390 Linux-s390x Mac-x86 Mac-x86_64 FreeBSD-x86_64 Windows-x86 Windows-x86_64 SunOS-x86 SunOS-sparc SunOS-x86_64 AIX-ppc AIX-ppc64
os_arch := $(OS_NAME)-$(OS_ARCH)
IBM_JDK_7 := $(findstring IBM, $(shell $(JAVA) -version 2>&1 | grep IBM | grep "JRE 1.7"))
@ -75,7 +75,7 @@ Default_SNAPPY_FLAGS :=
Linux-x86_CXX := $(CROSS_PREFIX)g++
Linux-x86_STRIP := $(CROSS_PREFIX)strip
ifeq ($(IBM_JDK_7),)
Linux-x86_CXXFLAGS := -include lib/inc_linux/jni_md.h -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -m32
Linux-x86_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -m32
else
Linux-x86_CXXFLAGS := -include $(IBM_JDK_LIB)/jni_md.h -include $(IBM_JDK_LIB)/jniport.h -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -m32
endif
@ -86,9 +86,9 @@ Linux-x86_SNAPPY_FLAGS:=
Linux-x86_64_CXX := $(CROSS_PREFIX)g++
Linux-x86_64_STRIP := $(CROSS_PREFIX)strip
ifeq ($(IBM_JDK_7),)
Linux-x86_64_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -Ilib/inc_mac -O2 -fPIC -fvisibility=hidden -m64
Linux-x86_64_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -m64
else
Linux-x86_64_CXXFLAGS := -include $(IBM_JDK_LIB)/jni_md.h -include $(IBM_JDK_LIB)/jniport.h -I$(JAVA_HOME)/include -Ilib/inc_mac -O2 -fPIC -fvisibility=hidden -m64
Linux-x86_64_CXXFLAGS := -include $(IBM_JDK_LIB)/jni_md.h -include $(IBM_JDK_LIB)/jniport.h -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -m64
endif
Linux-x86_64_LINKFLAGS := -shared -static-libgcc -static-libstdc++
Linux-x86_64_LIBNAME := libsnappyjava.so
@ -188,21 +188,35 @@ SunOS-x86_64_SNAPPY_FLAGS :=
Linux-arm_CXX := $(CROSS_PREFIX)g++
Linux-arm_STRIP := $(CROSS_PREFIX)strip
Linux-arm_CXXFLAGS := -include lib/inc_linux/jni_md.h -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -mfloat-abi=softfp
Linux-arm_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -mfloat-abi=softfp
Linux-arm_LINKFLAGS := -shared -static-libgcc
Linux-arm_LIBNAME := libsnappyjava.so
Linux-arm_SNAPPY_FLAGS:=
Linux-armhf_CXX := $(CROSS_PREFIX)g++
Linux-armhf_STRIP := $(CROSS_PREFIX)strip
Linux-armhf_CXXFLAGS := -include lib/inc_linux/jni_md.h -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -mfloat-abi=hard
Linux-armhf_LINKFLAGS := -shared -static-libgcc
Linux-armhf_LIBNAME := libsnappyjava.so
Linux-armhf_SNAPPY_FLAGS:=
Linux-armv6_CXX := $(CROSS_PREFIX)g++
Linux-armv6_STRIP := $(CROSS_PREFIX)strip
Linux-armv6_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -mfloat-abi=hard
Linux-armv6_LINKFLAGS := -shared -static-libgcc
Linux-armv6_LIBNAME := libsnappyjava.so
Linux-armv6_SNAPPY_FLAGS:=
Linux-armv7_CXX := $(CROSS_PREFIX)g++
Linux-armv7_STRIP := $(CROSS_PREFIX)strip
Linux-armv7_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden -mfloat-abi=hard
Linux-armv7_LINKFLAGS := -shared -static-libgcc
Linux-armv7_LIBNAME := libsnappyjava.so
Linux-armv7_SNAPPY_FLAGS:=
Linux-android-arm_CXX := $(CROSS_PREFIX)g++
Linux-android-arm_STRIP := $(CROSS_PREFIX)strip
Linux-android-arm_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden
Linux-android-arm_LINKFLAGS := -shared -static-libgcc
Linux-android-arm_LIBNAME := libsnappyjava.so
Linux-android-arm_SNAPPY_FLAGS:=
Linux-aarch64_CXX := $(CROSS_PREFIX)g++
Linux-aarch64_STRIP := $(CROSS_PREFIX)strip
Linux-aarch64_CXXFLAGS := -include lib/inc_linux/jni_md.h -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden
Linux-aarch64_CXXFLAGS := -Ilib/inc_linux -I$(JAVA_HOME)/include -O2 -fPIC -fvisibility=hidden
Linux-aarch64_LINKFLAGS := -shared -static-libgcc
Linux-aarch64_LIBNAME := libsnappyjava.so
Linux-aarch64_SNAPPY_FLAGS:=
@ -214,7 +228,7 @@ Mac-x86_LINKFLAGS := -dynamiclib -static-libgcc
Mac-x86_LIBNAME := libsnappyjava.jnilib
Mac-x86_SNAPPY_FLAGS :=
Mac-x86_64_CXX := g++ -arch $(OS_ARCH)
Mac-x86_64_CXX := c++ -arch $(OS_ARCH)
Mac-x86_64_STRIP := strip -x
Mac-x86_64_CXXFLAGS := -Ilib/inc_mac -I$(JAVA_HOME)/include -O2 -fPIC -mmacosx-version-min=10.5 -fvisibility=hidden
Mac-x86_64_LINKFLAGS := -dynamiclib

101
README.md
View File

@ -1,4 +1,6 @@
The snappy-java is a Java port of the snappy
snappy-java
===
snappy-java is a Java port of the snappy
<http://code.google.com/p/snappy/>, a fast C++ compresser/decompresser developed by Google.
## Features
@ -94,83 +96,11 @@ If you have snappy-java-(VERSION).jar in the current directory, use `-classpath`
$ javac -classpath ".:snappy-java-(VERSION).jar" Sample.java # in Mac or Linux
## Public discussion group
Post bug reports or feature request to the Issue Tracker: <https://github.com/xerial/snappy-java/issues>
Public discussion forum is here: [Xerial Public Discussion Group](http://groups.google.com/group/xerial?hl=en)
## Building from the source code
See the [installation instruction](https://github.com/xerial/snappy-java/blob/develop/INSTALL). Building from the source code is an option when your OS platform and CPU architecture is not supported. To build snappy-java, you need Git, JDK (1.6 or higher), g++ compiler (mingw in Windows) etc.
$ git clone https://github.com/xerial/snappy-java.git
$ cd snappy-java
$ make
When building on Solaris use
$ gmake
A file `target/snappy-java-$(version).jar` is the product additionally containing the native library built for your platform.
## Building Linux x86\_64 binary
snappy-java tries to static link libstdc++ to increase the availability for various Linux versions. However, standard distributions of 64-bit Linux OS rarely provide libstdc++ compiled with `-fPIC` option. I currently uses custom g++, compiled as follows:
```
$ cd work
$ wget (gcc-4.8.3 source)
$ tar xvfz (gcc-4.8.3.tar.gz)
$ cd gcc-4.8.3
$ ./contrib/download_prerequisites
$ cd ..
$ mkdir objdir
$ cd objdir
$ ../gcc-4.8.3/configure --prefix=$HOME/local/gcc-4.8.3 CXXFLAGS=-fPIC CFLAGS=-fPIC --enable-languages=c,c++
$ make
$ make install
```
This g++ build enables static linking of libstdc++. For more infomation on building GCC, see GCC's home page.
## Building Linux s390/s390x binaries
Older snapshots of snappy contain a buggy config.h.in that does not work properly on some big-endian platforms like Linux on IBM z (s390/s390x). Building snappy-java on s390/s390x requires fetching the snappy source from GitHub, and processing the source with autoconf to obtain a usable config.h. On a RHEL s390x system, these steps produced a working 64-bit snappy-java build (the process should be similar for other distributions):
$ sudo yum install java-1.7.1-ibm-devel libstdc++-static-devel
$ export JAVA_HOME=/usr/lib/jvm/java-1.7.1-ibm-1.7.1.2.10-1jpp.3.el7_0.s390x
$ make USE_GIT=1 GIT_REPO_URL=https://github.com/google/snappy.git GIT_SNAPPY_BRANCH=master IBM_JDK_7=1
## Activating SSE2/AVX2 instructions in BitShuffle
The most of the native libraries that snappy-java contains disable SSE2/AVX2 instructions in terms of portability (SSE2 is enabled only in Linux/x86_64 platforms). To enable AVX2 instructions, you need to compile as follows:
$ make CXXFLAGS_BITSHUFFLE=-mavx2 # -msse2 for SSE2 instructions
## Cross-compiling for other platforms
The Makefile contains rules for cross-compiling the native library for other platforms so that the snappy-java JAR can support multiple platforms. For example, to build the native libraries for x86 Linux, x86 and x86-64 Windows, and soft- and hard-float ARM:
$ make linux32 win32 win64 linux-arm linux-armhf linux-aarch64
If you append `snappy` to the line above, it will also build the native library for the current platform and then build the snappy-java JAR (containing all native libraries built so far).
Of course, you must first have the necessary cross-compilers and development libraries installed for each target CPU and OS. For example, on Ubuntu 12.04 for x86-64, install the following packages for each target:
* linux32: `sudo apt-get install g++-multilib libc6-dev-i386 lib32stdc++6`
* win32: `sudo apt-get install g++-mingw-w64-i686`
* win64: `sudo apt-get install g++-mingw-w64-x86-64`
* arm: `sudo apt-get install g++-arm-linux-gnueabi`
* armhf: `sudo apt-get install g++-arm-linux-gnueabihf`
* aarch64: `sudo apt-get install g++-aarch64-linux`
Unfortunately, cross-compiling for Mac OS X is not currently possible; you must compile within OS X.
If you are using Mac and openjdk7 (or higher), use the following option:
$ make native LIBNAME=libsnappyjava.dylib
## For developers
snappy-java uses sbt (simple build tool for Scala) as a build tool. Here is a simple usage
@ -191,10 +121,35 @@ $ ./sbt -Dloglevel=debug
For the details of sbt usage, see my blog post: [Building Java Projects with sbt](http://xerial.org/blog/2014/03/24/sbt/)
### Building from the source code
See the [build instruction](https://github.com/xerial/snappy-java/blob/master/BUILD.md). Building from the source code is an option when your OS platform and CPU architecture is not supported. To build snappy-java, you need Git, JDK (1.6 or higher), g++ compiler (mingw in Windows) etc.
$ git clone https://github.com/xerial/snappy-java.git
$ cd snappy-java
$ make
When building on Solaris, use `gmake`:
$ gmake
A file `target/snappy-java-$(version).jar` is the product additionally containing the native library built for your platform.
## Miscellaneous Notes
### Using snappy-java with Tomcat 6 (or higher) Web Server
Simply put the snappy-java's jar to WEB-INF/lib folder of your web application. Usual JNI-library specific problem no longer exists since snappy-java version 1.0.3 or higher can be loaded by multiple class loaders.
### Configure snappy-java using property file
Prepare org-xerial-snappy.properties file (under the root path of your library) in Java's property file format.
Here is a list of the available properties:
* org.xerial.snappy.lib.path (directory containing a snappyjava's native library)
* org.xerial.snappy.lib.name (library file name)
* org.xerial.snappy.tempdir (temporary directory to extract a native library bundled in snappy-java)
* org.xerial.snappy.use.systemlib (if this value is true, use system installed libsnappyjava.so looking the path specified by java.library.path)
----
Snappy-java is developed by [Taro L. Saito](http://www.xerial.org/leo). Twitter [@taroleo](http://twitter.com/#!/taroleo)

19
build.sbt
View File

@ -8,7 +8,16 @@ organizationName := "xerial.org"
description := "snappy-java: A fast compression/decompression library"
sonatypeProfileName := "org.xerial"
sonatypeProfileName := "org.xerial"
credentials ++= {
if(sys.env.contains("SONATYPE_USERNAME") && sys.env.contains("SONATYPE_PASSWORD")) {
Seq(Credentials("Sonatype Nexus Repository Manager", "oss.sonatype.org", sys.env("SONATYPE_USERNAME"), sys.env("SONATYPE_PASSWORD")))
}
else {
Seq.empty
}
}
pomExtra := {
<url>https://github.com/xerial/snappy-java</url>
@ -45,9 +54,9 @@ pomExtra := {
</scm>
}
scalaVersion := "2.11.6"
scalaVersion in ThisBuild := "2.11.8"
javacOptions in (Compile, compile) ++= Seq("-encoding", "UTF-8", "-Xlint:unchecked", "-Xlint:deprecation", "-source", "1.6", "-target", "1.6")
javacOptions in (Compile, compile) ++= Seq("-encoding", "UTF-8", "-Xlint:unchecked", "-Xlint:deprecation", "-source", "1.7", "-target", "1.7")
javacOptions in doc := {
val opts = Seq("-source", "1.6")
@ -112,8 +121,8 @@ OsgiKeys.additionalHeaders := Map(
"org/xerial/snappy/native/Linux/x86/libsnappyjava.so;osname=linux;processor=x86",
"org/xerial/snappy/native/Linux/aarch64/libsnappyjava.so;osname=linux;processor=aarch64",
"org/xerial/snappy/native/Linux/arm/libsnappyjava.so;osname=linux;processor=arm",
"org/xerial/snappy/native/Linux/ppc64/libsnappyjava.so;osname=linux;processor=ppc64",
"org/xerial/snappy/native/Linux/ppc64le/libsnappyjava.so;osname=linux;processor=ppc64le",
"org/xerial/snappy/native/Linux/arm7/libsnappyjava.so;osname=linux;processor=arm_le",
"org/xerial/snappy/native/Linux/ppc64/libsnappyjava.so;osname=linux;processor=ppc64le",
"org/xerial/snappy/native/Linux/s390x/libsnappyjava.so;osname=linux;processor=s390x",
"org/xerial/snappy/native/AIX/ppc/libsnappyjava.a;osname=aix;processor=ppc",
"org/xerial/snappy/native/AIX/ppc64/libsnappyjava.a;osname=aix;processor=ppc64",

27
docker/Dockerfile.centos5-linux-x86_64-pic Normal file
View File

@ -0,0 +1,27 @@
FROM centos:5
MAINTAINER Taro L. Saito <leo@xerial.org>
RUN yum -y install make gcc gcc-c++ glibc-devel perl wget bzip2 curl \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir /tmp/work \
&& cd /tmp/work \
&& wget http://www.netgull.com/gcc/releases/gcc-4.8.3/gcc-4.8.3.tar.gz \
&& tar xvfz gcc-4.8.3.tar.gz \
&& cd gcc-4.8.3 \
&& ./contrib/download_prerequisites \
&& cd .. \
&& mkdir objdir
RUN cd /tmp/work/objdir \
&& ../gcc-4.8.3/configure --prefix=/usr/local/gcc-4.8.3 CXXFLAGS=-fPIC CFLAGS=-fPIC --enable-languages=c,c++ \
&& make
RUN cd /tmp/work/objdir \
&& make install \
&& rm -rf /tmp/work
ENV PATH /usr/local/gcc-4.8.3/bin:$PATH
ENV LD_LIBRARY_PATH /usr/local/gcc-4.8.3/lib64/:$LD_LIBRARY_PATH
WORKDIR /work

200
docker/dockcross-aarch64 Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/linux-arm64
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

200
docker/dockcross-android-arm Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/android-arm
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

200
docker/dockcross-armv5 Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/linux-armv5
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

200
docker/dockcross-armv6 Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/linux-armv6
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

200
docker/dockcross-armv7 Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/linux-armv7
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

200
docker/dockcross-ppc64 Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/linux-ppc64le
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

200
docker/dockcross-windows-x64 Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/windows-x64
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

200
docker/dockcross-windows-x86 Executable file
View File

@ -0,0 +1,200 @@
#!/bin/bash
DEFAULT_DOCKCROSS_IMAGE=dockcross/windows-x86
#------------------------------------------------------------------------------
# Helpers
#
err() {
echo -e >&2 ERROR: $@\\n
}
die() {
err $@
exit 1
}
has() {
# eg. has command update
local kind=$1
local name=$2
type -t $kind:$name | grep -q function
}
#------------------------------------------------------------------------------
# Command handlers
#
command:update-image() {
docker pull $FINAL_IMAGE
}
help:update-image() {
echo Pull the latest $FINAL_IMAGE .
}
command:update-script() {
if cmp -s <( docker run $FINAL_IMAGE ) $0; then
echo $0 is up to date
else
echo -n Updating $0 '... '
docker run $FINAL_IMAGE > $0 && echo ok
fi
}
help:update-image() {
echo Update $0 from $FINAL_IMAGE .
}
command:update() {
command:update-image
command:update-script
}
help:update() {
echo Pull the latest $FINAL_IMAGE, and then update $0 from that.
}
command:help() {
if [[ $# != 0 ]]; then
if ! has command $1; then
err \"$1\" is not an dockcross command
command:help
elif ! has help $1; then
err No help found for \"$1\"
else
help:$1
fi
else
cat >&2 <<ENDHELP
Usage: dockcross [options] [--] command [args]
By default, run the given *command* in an dockcross Docker container.
The *options* can be one of:
--args|-a Extra args to the *docker run* command
--image|-i Docker cross-compiler image to use
--config|-c Bash script to source before running this script
Additionally, there are special update commands:
update-image
update-script
update
For update command help use: $0 help <command>
ENDHELP
exit 1
fi
}
#------------------------------------------------------------------------------
# Option processing
#
special_update_command=''
while [[ $# != 0 ]]; do
case $1 in
--)
break
;;
--args|-a)
ARG_ARGS="$2"
shift 2
;;
--config|-c)
ARG_CONFIG="$2"
shift 2
;;
--image|-i)
ARG_IMAGE="$2"
shift 2
;;
update|update-image|update-script)
special_update_command=$1
break
;;
-*)
err Unknown option \"$1\"
command:help
exit
;;
*)
break
;;
esac
done
# The precedence for options is:
# 1. command-line arguments
# 2. environment variables
# 3. defaults
# Source the config file if it exists
DEFAULT_DOCKCROSS_CONFIG=~/.dockcross
FINAL_CONFIG=${ARG_CONFIG-${DOCKCROSS_CONFIG-$DEFAULT_DOCKCROSS_CONFIG}}
[[ -f "$FINAL_CONFIG" ]] && source "$FINAL_CONFIG"
# Set the docker image
FINAL_IMAGE=${ARG_IMAGE-${DOCKCROSS_IMAGE-$DEFAULT_DOCKCROSS_IMAGE}}
# Handle special update command
if [ "$special_update_command" != "" ]; then
case $special_update_command in
update)
command:update
exit $?
;;
update-image)
command:update-image
exit $?
;;
update-script)
command:update-script
exit $?
;;
esac
fi
# Set the docker run extra args (if any)
FINAL_ARGS=${ARG_ARGS-${DOCKCROSS_ARGS}}
# If we are not running via boot2docker
if [ -z $DOCKER_HOST ]; then
USER_IDS="-e BUILDER_UID=$( id -u ) -e BUILDER_GID=$( id -g ) -e BUILDER_USER=$( id -un ) -e BUILDER_GROUP=$( id -gn )"
fi
#------------------------------------------------------------------------------
# Now, finally, run the command in a container
#
docker run --rm \
-v $PWD:/work \
$USER_IDS \
$FINAL_ARGS \
$FINAL_IMAGE "$@"
################################################################################
#
# This image is not intended to be run manually.
#
# To create a dockcross helper script for the
# dockcross/linux-armv7 image, run:
#
# docker run --rm dockcross/linux-armv7 > dockcross-linux-armv7
# chmod +x dockcross-linux-armv7
#
# You may then wish to move the dockcross script to your PATH.
#
################################################################################

248
googlecode_upload.py
View File

@ -1,248 +0,0 @@
#!/usr/bin/env python
#
# Copyright 2006, 2007 Google Inc. All Rights Reserved.
# Author: danderson@google.com (David Anderson)
#
# Script for uploading files to a Google Code project.
#
# This is intended to be both a useful script for people who want to
# streamline project uploads and a reference implementation for
# uploading files to Google Code projects.
#
# To upload a file to Google Code, you need to provide a path to the
# file on your local machine, a small summary of what the file is, a
# project name, and a valid account that is a member or owner of that
# project. You can optionally provide a list of labels that apply to
# the file. The file will be uploaded under the same name that it has
# in your local filesystem (that is, the "basename" or last path
# component). Run the script with '--help' to get the exact syntax
# and available options.
#
# Note that the upload script requests that you enter your
# googlecode.com password. This is NOT your Gmail account password!
# This is the password you use on googlecode.com for committing to
# Subversion and uploading files. You can find your password by going
# to http://code.google.com/hosting/settings when logged in with your
# Gmail account. If you have already committed to your project's
# Subversion repository, the script will automatically retrieve your
# credentials from there (unless disabled, see the output of '--help'
# for details).
#
# If you are looking at this script as a reference for implementing
# your own Google Code file uploader, then you should take a look at
# the upload() function, which is the meat of the uploader. You
# basically need to build a multipart/form-data POST request with the
# right fields and send it to https://PROJECT.googlecode.com/files .
# Authenticate the request using HTTP Basic authentication, as is
# shown below.
#
# Licensed under the terms of the Apache Software License 2.0:
# http://www.apache.org/licenses/LICENSE-2.0
#
# Questions, comments, feature requests and patches are most welcome.
# Please direct all of these to the Google Code users group:
# http://groups.google.com/group/google-code-hosting
"""Google Code file uploader script.
"""
__author__ = 'danderson@google.com (David Anderson)'
import httplib
import os.path
import optparse
import getpass
import base64
import sys
def upload(file, project_name, user_name, password, summary, labels=None):
"""Upload a file to a Google Code project's file server.
Args:
file: The local path to the file.
project_name: The name of your project on Google Code.
user_name: Your Google account name.
password: The googlecode.com password for your account.
Note that this is NOT your global Google Account password!
summary: A small description for the file.
labels: an optional list of label strings with which to tag the file.
Returns: a tuple:
http_status: 201 if the upload succeeded, something else if an
error occured.
http_reason: The human-readable string associated with http_status
file_url: If the upload succeeded, the URL of the file on Google
Code, None otherwise.
"""
# The login is the user part of user@gmail.com. If the login provided
# is in the full user@domain form, strip it down.
if user_name.endswith('@gmail.com'):
user_name = user_name[:user_name.index('@gmail.com')]
form_fields = [('summary', summary)]
if labels is not None:
form_fields.extend([('label', l.strip()) for l in labels])
content_type, body = encode_upload_request(form_fields, file)
upload_host = '%s.googlecode.com' % project_name
upload_uri = '/files'
auth_token = base64.b64encode('%s:%s'% (user_name, password))
headers = {
'Authorization': 'Basic %s' % auth_token,
'User-Agent': 'Googlecode.com uploader v0.9.4',
'Content-Type': content_type,
}
server = httplib.HTTPSConnection(upload_host)
server.request('POST', upload_uri, body, headers)
resp = server.getresponse()
server.close()
if resp.status == 201:
location = resp.getheader('Location', None)
else:
location = None
return resp.status, resp.reason, location
def encode_upload_request(fields, file_path):
"""Encode the given fields and file into a multipart form body.
fields is a sequence of (name, value) pairs. file is the path of
the file to upload. The file will be uploaded to Google Code with
the same file name.
Returns: (content_type, body) ready for httplib.HTTP instance
"""
BOUNDARY = '----------Googlecode_boundary_reindeer_flotilla'
CRLF = '\r\n'
body = []
# Add the metadata about the upload first
for key, value in fields:
body.extend(
['--' + BOUNDARY,
'Content-Disposition: form-data; name="%s"' % key,
'',
value,
])
# Now add the file itself
file_name = os.path.basename(file_path)
f = open(file_path, 'rb')
file_content = f.read()
f.close()
body.extend(
['--' + BOUNDARY,
'Content-Disposition: form-data; name="filename"; filename="%s"'
% file_name,
# The upload server determines the mime-type, no need to set it.
'Content-Type: application/octet-stream',
'',
file_content,
])
# Finalize the form body
body.extend(['--' + BOUNDARY + '--', ''])
return 'multipart/form-data; boundary=%s' % BOUNDARY, CRLF.join(body)
def upload_find_auth(file_path, project_name, summary, labels=None,
user_name=None, password=None, tries=3):
"""Find credentials and upload a file to a Google Code project's file server.
file_path, project_name, summary, and labels are passed as-is to upload.
Args:
file_path: The local path to the file.
project_name: The name of your project on Google Code.
summary: A small description for the file.
labels: an optional list of label strings with which to tag the file.
config_dir: Path to Subversion configuration directory, 'none', or None.
user_name: Your Google account name.
tries: How many attempts to make.
"""
while tries > 0:
if user_name is None:
# Read username if not specified or loaded from svn config, or on
# subsequent tries.
sys.stdout.write('Please enter your googlecode.com username: ')
sys.stdout.flush()
user_name = sys.stdin.readline().rstrip()
if password is None:
# Read password if not loaded from svn config, or on subsequent tries.
print 'Please enter your googlecode.com password.'
print '** Note that this is NOT your Gmail account password! **'
print 'It is the password you use to access Subversion repositories,'
print 'and can be found here: http://code.google.com/hosting/settings'
password = getpass.getpass()
status, reason, url = upload(file_path, project_name, user_name, password,
summary, labels)
# Returns 403 Forbidden instead of 401 Unauthorized for bad
# credentials as of 2007-07-17.
if status in [httplib.FORBIDDEN, httplib.UNAUTHORIZED]:
# Rest for another try.
user_name = password = None
tries = tries - 1
else:
# We're done.
break
return status, reason, url
def main():
parser = optparse.OptionParser(usage='googlecode-upload.py -s SUMMARY '
'-p PROJECT [options] FILE')
parser.add_option('-s', '--summary', dest='summary',
help='Short description of the file')
parser.add_option('-p', '--project', dest='project',
help='Google Code project name')
parser.add_option('-u', '--user', dest='user',
help='Your Google Code username')
parser.add_option('-w', '--password', dest='password',
help='Your Google Code password')
parser.add_option('-l', '--labels', dest='labels',
help='An optional list of comma-separated labels to attach '
'to the file')
options, args = parser.parse_args()
if not options.summary:
parser.error('File summary is missing.')
elif not options.project:
parser.error('Project name is missing.')
elif len(args) < 1:
parser.error('File to upload not provided.')
elif len(args) > 1:
parser.error('Only one file may be specified.')
file_path = args[0]
if options.labels:
labels = options.labels.split(',')
else:
labels = None
status, reason, url = upload_find_auth(file_path, options.project,
options.summary, labels,
options.user, options.password)
if url:
print 'The file was uploaded successfully.'
print 'URL: %s' % url
return 0
else:
print 'An error occurred. Your file was not uploaded.'
print 'Google Code upload server said: %s (%s)' % (reason, status)
return 1
if __name__ == '__main__':
sys.exit(main())

BIN
lib/org/xerial/snappy/OSInfo.class
View File

Binary file not shown.

2
project/build.properties
View File

@ -1,2 +1,2 @@
sbt.version=0.13.9
sbt.version=0.13.13

10
project/plugins.sbt
View File

@ -1,14 +1,8 @@
addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.0")
addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.3")
addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "1.1")
addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.0.0")
addSbtPlugin("de.johoop" % "findbugs4sbt" % "1.4.0")
addSbtPlugin("de.johoop" % "jacoco4sbt" % "2.1.5")
addSbtPlugin("com.typesafe.sbt" % "sbt-osgi" % "0.7.0")
addSbtPlugin("com.etsy" % "sbt-checkstyle-plugin" % "0.4.3")
addSbtPlugin("io.get-coursier" % "sbt-coursier" % "1.0.0-M15")

2
sbt
View File

@ -101,7 +101,7 @@ init_default_option_file () {
declare -r cms_opts="-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC"
declare -r jit_opts="-XX:ReservedCodeCacheSize=256m -XX:+TieredCompilation"
declare -r default_jvm_opts_common="-Xms512m -Xmx1536m -Xss2m $jit_opts $cms_opts"
declare -r default_jvm_opts_common="-Xms512m -Xmx2048m -Xss2m $jit_opts $cms_opts"
declare -r noshare_opts="-Dsbt.global.base=project/.sbtboot -Dsbt.boot.directory=project/.boot -Dsbt.ivy.home=project/.ivy"
declare -r latest_28="2.8.2"
declare -r latest_29="2.9.3"

136
src/main/java/org/xerial/snappy/OSInfo.java
View File

@ -24,7 +24,9 @@
//--------------------------------------
package org.xerial.snappy;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.HashMap;
import java.util.Locale;
@ -33,8 +35,7 @@ import java.util.Locale;
*
* @author leo
*/
public class OSInfo
{
public class OSInfo {
private static HashMap<String, String> archMapping = new HashMap<String, String>();
public static final String X86 = "x86";
@ -94,14 +95,13 @@ public class OSInfo
archMapping.put(AARCH_64, AARCH_64);
}
public static void main(String[] args)
{
if (args.length >= 1) {
if ("--os".equals(args[0])) {
public static void main(String[] args) {
if(args.length >= 1) {
if("--os".equals(args[0])) {
System.out.print(getOSName());
return;
}
else if ("--arch".equals(args[0])) {
else if("--arch".equals(args[0])) {
System.out.print(getArchName());
return;
}
@ -110,60 +110,125 @@ public class OSInfo
System.out.print(getNativeLibFolderPathForCurrentOS());
}
public static String getNativeLibFolderPathForCurrentOS()
{
public static String getNativeLibFolderPathForCurrentOS() {
return getOSName() + "/" + getArchName();
}
public static String getOSName()
{
public static String getOSName() {
return translateOSNameToFolderName(System.getProperty("os.name"));
}
public static String getArchName()
{
// if running Linux on ARM, need to determine ABI of JVM
String osArch = System.getProperty("os.arch");
if (osArch.startsWith("arm") && System.getProperty("os.name").contains("Linux")) {
String javaHome = System.getProperty("java.home");
public static boolean isAndroid() {
return System.getProperty("java.runtime.name", "").toLowerCase().contains("android");
}
static String getHardwareName() {
try {
Process p = Runtime.getRuntime().exec("uname -m");
p.waitFor();
InputStream in = p.getInputStream();
try {
// determine if first JVM found uses ARM hard-float ABI
String[] cmdarray = {"/bin/sh", "-c", "find '" + javaHome +
"' -name 'libjvm.so' | head -1 | xargs readelf -A | " +
"grep 'Tag_ABI_VFP_args: VFP registers'"};
int exitCode = Runtime.getRuntime().exec(cmdarray).waitFor();
if (exitCode == 0) {
return "armhf";
int readLen = 0;
ByteArrayOutputStream b = new ByteArrayOutputStream();
byte[] buf = new byte[32];
while((readLen = in.read(buf, 0, buf.length)) >= 0) {
b.write(buf, 0, readLen);
}
return b.toString();
}
finally {
if(in != null) {
in.close();
}
}
catch (IOException e) {
}
catch(Throwable e) {
System.err.println("Error while running uname -m: " + e.getMessage());
return "unknown";
}
}
static String resolveArmArchType() {
// For Android
if(isAndroid()) {
return "android-arm";
}
if(System.getProperty("os.name").contains("Linux")) {
String armType = getHardwareName();
// armType (uname -m) can be armv5t, armv5te, armv5tej, armv5tejl, armv6, armv7, armv7l, i686
if(armType.startsWith("armv6")) {
// Raspberry PI
return "armv6";
}
else if(armType.startsWith("armv7")) {
// Generic
return "armv7";
}
// Java 1.8 introduces a system property to determine armel or armhf
// http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8005545
String abi = System.getProperty("sun.arch.abi");
if(abi != null && abi.startsWith("gnueabihf")) {
return "armv7";
}
// For java7, we stil need to if run some shell commands to determine ABI of JVM
try {
// determine if first JVM found uses ARM hard-float ABI
int exitCode = Runtime.getRuntime().exec("which readelf").waitFor();
if(exitCode == 0) {
String javaHome = System.getProperty("java.home");
String[] cmdarray = {"/bin/sh", "-c", "find '" + javaHome +
"' -name 'libjvm.so' | head -1 | xargs readelf -A | " +
"grep 'Tag_ABI_VFP_args: VFP registers'"};
exitCode = Runtime.getRuntime().exec(cmdarray).waitFor();
if(exitCode == 0) {
return "armv7";
}
}
else {
System.err.println("WARNING! readelf not found. Cannot check if running on an armhf system, " +
"armel architecture will be presumed.");
}
}
catch(IOException e) {
// ignored: fall back to "arm" arch (soft-float ABI)
}
catch (InterruptedException e) {
catch(InterruptedException e) {
// ignored: fall back to "arm" arch (soft-float ABI)
}
}
// Use armv5, soft-float ABI
return "arm";
}
public static String getArchName() {
String osArch = System.getProperty("os.arch");
if(osArch.startsWith("arm")) {
osArch = resolveArmArchType();
}
else {
String lc = osArch.toLowerCase(Locale.US);
if (archMapping.containsKey(lc)) {
if(archMapping.containsKey(lc))
return archMapping.get(lc);
}
}
return translateArchNameToFolderName(osArch);
}
static String translateOSNameToFolderName(String osName)
{
if (osName.contains("Windows")) {
static String translateOSNameToFolderName(String osName) {
if(osName.contains("Windows")) {
return "Windows";
}
else if (osName.contains("Mac")) {
else if(osName.contains("Mac")) {
return "Mac";
}
else if (osName.contains("Linux")) {
else if(osName.contains("Linux")) {
return "Linux";
}
else if (osName.contains("AIX")) {
else if(osName.contains("AIX")) {
return "AIX";
}
else {
@ -171,8 +236,7 @@ public class OSInfo
}
}
static String translateArchNameToFolderName(String archName)
{
static String translateArchNameToFolderName(String archName) {
return archName.replaceAll("\\W", "");
}
}

BIN
src/main/resources/org/xerial/snappy/native/Linux/aarch64/libsnappyjava.so
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/android-arm/libsnappyjava.so Executable file
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/arm/libsnappyjava.so
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/armhf/libsnappyjava.so
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/armv6/libsnappyjava.so Executable file
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/armv7/libsnappyjava.so Executable file
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/ppc64/libsnappyjava.so
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/ppc64le/libsnappyjava.so
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/x86/libsnappyjava.so
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Mac/x86_64/libsnappyjava.jnilib
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Windows/x86/snappyjava.dll
View File

Binary file not shown.

BIN
src/main/resources/org/xerial/snappy/native/Windows/x86_64/snappyjava.dll
View File

Binary file not shown.

4
src/test/java/org/xerial/snappy/SnappyOutputStreamTest.java
View File

@ -166,9 +166,9 @@ public class SnappyOutputStreamTest
// Hardcoding an expected compressed size here will catch regressions that lower the
// compression quality:
if (ByteOrder.nativeOrder() == ByteOrder.BIG_ENDIAN)
assertEquals(90943, expectedCompressedData.length);
assertEquals(91080, expectedCompressedData.length);
else
assertEquals(91013, expectedCompressedData.length);
assertEquals(91080, expectedCompressedData.length);
// The chunk size should not affect the size of the compressed output:
int[] chunkSizes = new int[] {1, 100, 1023, 1024, 10000};
for (int chunkSize : chunkSizes) {