[WIP] Building Apache Hadoop with Apple Silicon
Install Java
I used AdoptOpenJDK 8. The binary is for x64 but somehow it worked when building.
When running tests, Kerberos does not work:
Stack: [0x000000030b6b4000,0x000000030b7b4000], sp=0x000000030b7b1730, free space=1013k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [Heimdal+0xff9c] HeimCredGetUUID+0xc C [Heimdal+0x21dc1] xcc_get_first+0x41 C [Heimdal+0x159bd] krb5_cc_start_seq_get+0x2d C [libosxkrb5.dylib+0x1a01] Java_sun_security_krb5_Credentials_acquireDefaultNativeCreds+0x87 j sun.security.krb5.Credentials.acquireDefaultNativeCreds([I)Lsun/security/krb5/Credentials;+0 j sun.security.krb5.Credentials.acquireDefaultCreds()Lsun/security/krb5/Credentials;+164 j sun.security.krb5.Credentials.acquireTGTFromCache(Lsun/security/krb5/PrincipalName;Ljava/lang/String;)Lsun/security/krb5/Credentials;+50 j com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Z)V+81 j com.sun.security.auth.module.Krb5LoginModule.login()Z+280
Inet4AddressImpl.lookupAllHostAddr hangs:
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
-------------------------------------------------------------------------------
Tests run: 14, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 73.992 s <<< FAILURE! - in org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
testHangOnSinkRead(org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl) Time elapsed: 5.114 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 5000 milliseconds
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getLocalHost(InetAddress.java:1501)
Probably I need to build JDK for Apple Silicon.
Building Java code
Failed to build hadoop-yarn-application-catalog
Old version of node does not support ARM architecture. Edit the nodeVersion to the latest version in the pom.xml worked. I'll file a JIRA.
Building native code
Install some homebrew
Homebrew does not work without workaround. The workaround and supported packages are in the page: macOS 10.16/11.0 Big Sur compatibility · Issue #7857 · Homebrew/brew · GitHub
I installed Doxygen, protobuf, protobuf-c, and openssl@1.1 via homebrew. Openssl is not installed in the normal path (because MacOS already have LibreSSL), so I had to set the following option:
-Dopenssl.prefix=/usr/local/opt/openssl@1.1
Install zlib
Downloaded source and built it. (Probably I didn't have to do this because zlib can be installed via homebrew)
Install boost 1.72
Downloaded source and built it. Now Apache Hadoop support boost 1.72 only. Do not download the latest version.
Install CMake
Downloaded the latest version and built it.
Failed to build hadoop-hdfs-native-client
[WARNING] [ 10%] Built target hdfs_static [WARNING] [ 10%] Built target hdfs [WARNING] ld: warning: ignoring file /usr/local/lib/libprotoc.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64 [WARNING] ld: warning: ignoring file /usr/local/lib/libprotobuf.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64 [WARNING] Undefined symbols for architecture x86_64: [WARNING] "google::protobuf::io::Printer::Print(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&, char const*)", referenced from: [WARNING] StubGenerator::EmitService(google::protobuf::ServiceDescriptor const*, google::protobuf::io::Printer*) const in protoc_gen_hrpc.cc.o [WARNING] void google::protobuf::io::Printer::PrintInternal<>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >*, char const*, char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in protoc_gen_hrpc.cc.o [WARNING] "google::protobuf::io::Printer::Printer(google::protobuf::io::ZeroCopyOutputStream*, char)", referenced from: [WARNING] StubGenerator::Generate(google::protobuf::FileDescriptor const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::compiler::GeneratorContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) const in protoc_gen_hrpc.cc.o [WARNING] "google::protobuf::io::Printer::~Printer()", referenced from: [WARNING] StubGenerator::Generate(google::protobuf::FileDescriptor const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::compiler::GeneratorContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) const in protoc_gen_hrpc.cc.o [WARNING] "google::protobuf::compiler::PluginMain(int, char**, google::protobuf::compiler::CodeGenerator const*)", referenced from: [WARNING] _main in protoc_gen_hrpc.cc.o [WARNING] "google::protobuf::compiler::CodeGenerator::~CodeGenerator()", referenced from: [WARNING] _main in protoc_gen_hrpc.cc.o [WARNING] StubGenerator::~StubGenerator() in protoc_gen_hrpc.cc.o [WARNING] StubGenerator::~StubGenerator() in protoc_gen_hrpc.cc.o [WARNING] "google::protobuf::compiler::CodeGenerator::GenerateAll(std::__1::vector<google::protobuf::FileDescriptor const*, std::__1::allocator<google::protobuf::FileDescriptor const*> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::compiler::GeneratorContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) const", referenced from: [WARNING] vtable for StubGenerator in protoc_gen_hrpc.cc.o [WARNING] "typeinfo for google::protobuf::compiler::CodeGenerator", referenced from: [WARNING] typeinfo for StubGenerator in protoc_gen_hrpc.cc.o [WARNING] ld: symbol(s) not found for architecture x86_64 [WARNING] clang: error: linker command failed with exit code 1 (use -v to see invocation) [WARNING] make[2]: *** [main/native/libhdfspp/lib/proto/protoc-gen-hrpc] Error 1 [WARNING] make[1]: *** [main/native/libhdfspp/lib/proto/CMakeFiles/protoc-gen-hrpc.dir/all] Error 2 [WARNING] make[1]: *** Waiting for unfinished jobs.... [WARNING] make: *** [all] Error 2 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 19.283 s [INFO] Finished at: 2020-07-24T01:26:40+09:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.4.0-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-hdfs-native-client: make failed with error code 2 -> [Help 1]
:thinking_face:
