Apache Sparkは最近一気に脚光を浴び始めた。

IBMが何千者研究者を投入するニュースからそんなに時間が立ってないうち、

「BigInsights for Apache Hadoop V4.1」にSparkが使えるようになった。
Sparkの技術だけをマスターできれば、将来はご飯を食べていけるのではないか？
冗談ですけど、寝る前にちょっと勉強したいです。
★ダウンロード
apacheのホームページから、mavenとsparkのパッケージをダウンロードしてきます。
・apache-maven-3.3.3-bin.tar.gz
・spark-1.4.0.tgz（パッケージタイプ：Source Code）
↑ソースコードタイプを使って、自分でビルドすると、Sparkだけではなく、サンプルもダウンロードできます。

★mavenのセットアップは割愛。

解凍して配置して、PATHを通せばOK。

解凍コマンド：tar zxvf apache-maven-3.3.3-bin.tar.gz

$ mvn -version
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T20:57:37+09:00)
Maven home: /usr/local/apache-maven-3.3.3
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: ja_JP, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-504.23.4.el6.x86_64", arch: "amd64", family: "unix"

$mvn -DskipTests clean package
…
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [03:00 min]
[INFO] Spark Launcher Project ............................. SUCCESS [01:11 min]
[INFO] Spark Project Networking ........................... SUCCESS [ 11.874 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 5.619 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 4.642 s]
[INFO] Spark Project Core ................................. SUCCESS [04:07 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 17.762 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 57.769 s]
[INFO] Spark Project Streaming ............................ SUCCESS [01:18 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:26 min]
[INFO] Spark Project SQL .................................. SUCCESS [01:56 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:20 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 10.926 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:47 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 35.895 s]
[INFO] Spark Project Assembly ............................. SUCCESS [01:06 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 21.557 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 26.509 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 21.713 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [ 47.694 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 16.291 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [ 31.361 s]
[INFO] Spark Project Examples ............................. SUCCESS [01:55 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 19.984 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25:54 min
[INFO] Finished at: 2015-07-14T21:30:39+09:00
[INFO] Final Memory: 81M/980M
[INFO] ------------------------------------------------------------------------

→光フレッツの回線でやく２６分でダウンロードとビルドできた。

$ ./bin/spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/14 21:35:07 INFO SecurityManager: Changing view acls to: jeff
15/07/14 21:35:07 INFO SecurityManager: Changing modify acls to: jeff
15/07/14 21:35:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jeff); users with modify permissions: Set(jeff)
15/07/14 21:35:07 INFO HttpServer: Starting HTTP Server
15/07/14 21:35:07 INFO Utils: Successfully started service 'HTTP class server' on port 52125.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45)
Type in expressions to have them evaluated.

Type :help for more information.
…
scala> sc.parallelize(1 to 1000).count()
<console>:14: error: not found: value sc
sc.parallelize(1 to 1000).count()
^
なんかエラーが出た。時間がないので、今日は寝る。

時間があるとき、また原因探し。。。

ごめん、こんなところで止めた。(-_-)zzz

------------------------------
以下は５日間後に作業再開（遅すぎるけど、ごめん、休日で暇な時しかやってない）

↑前回の続きで、原因探し
エラーメッセージの一部は以下となっていた。（ほとんど以下エラーの繰り返し）
15/07/19 10:22:23 INFO Slf4jLogger: Slf4jLogger started
15/07/19 10:22:23 INFO Remoting: Starting remoting
15/07/19 10:22:26 ERROR NettyTransport: failed to bind to /203.189.105.208:0, shutting down Netty transport
15/07/19 10:22:26 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
15/07/19 10:22:26 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/07/19 10:22:26 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/07/19 10:22:26 INFO Slf4jLogger: Slf4jLogger started
15/07/19 10:22:26 INFO Remoting: Starting remoting

★この原因としては、マシンのホスト名を***.comと設定してしまったせいのようです。
　***.comのホスト名は実は別にあった。（うちの場合自分が所有しているものだった）
参考サイトは下記でした。

http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac

→see Answer2

マシンのホスト名を***.com →　***.localに変更して再起動する。

★今度は、下記エラーとなった。
Type :help for more information.
java.net.UnknownHostException: ***.local: ***.local: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1484)

ホスト名見つからないから、たぶん/etc/hostsの設定をしてなかったためだ。
以下のように設定を追加した。

192.168.xxx.yyy *** ***.local

すると、再度実行したら、エラーが消えて正常に起動できた。

$ ./bin/spark-shell
…
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
15/07/19 10:48:27 INFO SparkContext: Running Spark version 1.4.0
…
scala>

★以下はscalaと付属してきたREADME.md（テキストファイル）を使った動作確認。
　（１）README.mdの行数カウント
　（２）README.mdの最初行の文字列を出力

scala> val lines=sc.textFile("README.md")
15/07/19 10:55:34 INFO MemoryStore: ensureFreeSpace(110248) called with curMem=1900, maxMem=278019440
15/07/19 10:55:34 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 107.7 KB, free 265.0 MB)
15/07/19 10:55:34 INFO MemoryStore: ensureFreeSpace(10090) called with curMem=112148, maxMem=278019440
15/07/19 10:55:34 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 9.9 KB, free 265.0 MB)
15/07/19 10:55:34 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:34144 (size: 9.9 KB, free: 265.1 MB)
15/07/19 10:55:34 INFO SparkContext: Created broadcast 1 from textFile at <console>:21
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at textFile at <console>:21

scala> lines.count()
15/07/19 10:58:20 INFO FileInputFormat: Total input paths to process : 1
15/07/19 10:58:20 INFO SparkContext: Starting job: count at <console>:24
15/07/19 10:58:20 INFO DAGScheduler: Got job 1 (count at <console>:24) with 2 output partitions (allowLocal=false)
15/07/19 10:58:20 INFO DAGScheduler: Final stage: ResultStage 1(count at <console>:24)
15/07/19 10:58:20 INFO DAGScheduler: Parents of final stage: List()
15/07/19 10:58:20 INFO DAGScheduler: Missing parents: List()
15/07/19 10:58:20 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[2] at textFile at <console>:21), which has no missing parents
15/07/19 10:58:20 INFO MemoryStore: ensureFreeSpace(2960) called with curMem=122238, maxMem=278019440
15/07/19 10:58:20 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.9 KB, free 265.0 MB)
15/07/19 10:58:20 INFO MemoryStore: ensureFreeSpace(1745) called with curMem=125198, maxMem=278019440
15/07/19 10:58:20 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1745.0 B, free 265.0 MB)
15/07/19 10:58:20 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:34144 (size: 1745.0 B, free: 265.1 MB)
15/07/19 10:58:20 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874
15/07/19 10:58:20 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[2] at textFile at <console>:21)
15/07/19 10:58:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
15/07/19 10:58:20 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, PROCESS_LOCAL, 1410 bytes)
15/07/19 10:58:20 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, localhost, PROCESS_LOCAL, 1410 bytes)
15/07/19 10:58:20 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
15/07/19 10:58:20 INFO Executor: Running task 1.0 in stage 1.0 (TID 3)
15/07/19 10:58:20 INFO HadoopRDD: Input split: file:/home/jeff/spark-1.4.0/README.md:0+1812
15/07/19 10:58:20 INFO HadoopRDD: Input split: file:/home/jeff/spark-1.4.0/README.md:1812+1812
15/07/19 10:58:20 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
15/07/19 10:58:20 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/07/19 10:58:20 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/07/19 10:58:20 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/07/19 10:58:20 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/07/19 10:58:20 INFO Executor: Finished task 1.0 in stage 1.0 (TID 3). 1830 bytes result sent to driver
15/07/19 10:58:20 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 1830 bytes result sent to driver
15/07/19 10:58:20 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 88 ms on localhost (1/2)
15/07/19 10:58:20 INFO DAGScheduler: ResultStage 1 (count at <console>:24) finished in 0.088 s
15/07/19 10:58:20 INFO DAGScheduler: Job 1 finished: count at <console>:24, took 0.136062 s
res1: Long = 98

scala> 15/07/19 10:58:20 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 89 ms on localhost (2/2)
15/07/19 10:58:20 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool

scala> lines.first()
15/07/19 10:59:40 INFO SparkContext: Starting job: first at <console>:24
15/07/19 10:59:40 INFO DAGScheduler: Got job 2 (first at <console>:24) with 1 output partitions (allowLocal=true)
15/07/19 10:59:40 INFO DAGScheduler: Final stage: ResultStage 2(first at <console>:24)
15/07/19 10:59:40 INFO DAGScheduler: Parents of final stage: List()
15/07/19 10:59:40 INFO DAGScheduler: Missing parents: List()
15/07/19 10:59:40 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[2] at textFile at <console>:21), which has no missing parents
15/07/19 10:59:40 INFO MemoryStore: ensureFreeSpace(3128) called with curMem=126943, maxMem=278019440
15/07/19 10:59:40 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.1 KB, free 265.0 MB)
15/07/19 10:59:40 INFO MemoryStore: ensureFreeSpace(1805) called with curMem=130071, maxMem=278019440
15/07/19 10:59:40 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1805.0 B, free 265.0 MB)
15/07/19 10:59:40 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:34144 (size: 1805.0 B, free: 265.1 MB)
15/07/19 10:59:40 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:874
15/07/19 10:59:40 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[2] at textFile at <console>:21)
15/07/19 10:59:40 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
15/07/19 10:59:40 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, localhost, PROCESS_LOCAL, 1410 bytes)
15/07/19 10:59:40 INFO Executor: Running task 0.0 in stage 2.0 (TID 4)
15/07/19 10:59:40 INFO HadoopRDD: Input split: file:/home/jeff/spark-1.4.0/README.md:0+1812
15/07/19 10:59:40 INFO Executor: Finished task 0.0 in stage 2.0 (TID 4). 1809 bytes result sent to driver
15/07/19 10:59:40 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 18 ms on localhost (1/1)
15/07/19 10:59:40 INFO DAGScheduler: ResultStage 2 (first at <console>:24) finished in 0.018 s
15/07/19 10:59:40 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
15/07/19 10:59:40 INFO DAGScheduler: Job 2 finished: first at <console>:24, took 0.029939 s
res2: String = # Apache Spark

★実際、上記のようにINFOメッセージがいっぱい出力されることを抑止することができる。
設定ファイル：conf/log4j.properties
設定方法：
$ cp conf/log4j.properties.template conf/log4j.properties
$ vi conf/log4j.propertie
…
#log4j.rootCategory=INFO, console
log4j.rootCategory=WARN, console
…
保存して終了

★上記の動作確認を再度実行すると、以下のように変わった。
[jeff@liandus spark-1.4.0]$ ./bin/spark-shell
15/07/19 11:10:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
SQL context available as sqlContext.

scala> val lines=sc.textFile("README.md")
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:21

scala> lines.count()
res0: Long = 98

scala> lines.first()
res1: String = # Apache Spark

NHocr: OCR engine for Japanese language

http://osdn.jp/projects/nhocr/releases/

上記アドレスから、下記をダウンロードする

・nhocr-0.22.tar.gz

nhocr-0.22/READMEの説明通り、

$./configure

$make

…

g++ -I../libnhocr -I../include -O3 --unroll-loops -o makedic makedic.o -L../libnhocr -L../O2-tools/lib -lm -lnhocr -lufp -lsip

make[1]: ディレクトリ `/home/jeff/nhocr-0.22/makedic' から出ます

make[1]: ディレクトリ `/home/jeff/nhocr-0.22' に入ります

make[1]: `all-am' に対して行うべき事はありません.

make[1]: ディレクトリ `/home/jeff/nhocr-0.22' から出ます

# ls -l nhocr/

…

-rwxr-xr-x. 1 root root 261083 7月 9 23:51 2015 nhocr

…

上記が生成すればOK

事前に準備したjpgファイルをpgm形式に変換する。

# convert receipt.jpg receipt.pgm

下記のように、コマンドを実行すれば、写真の中の文字を識別できる。

# NHOCR_DICDIR=dic nhocr/nhocr -block -o output.txt receipt.pgm

参考情報

※make時に下記のようなエラーが発生した場合、

--------------------------------------------

makechar.cpp:18:22: error: ft2build.h: そのようなファイルやディレクトリはありません

makechar.cpp:19:10: error: #include は "ファイル名" あるいは <ファイル名> を必要とします

makechar.cpp:85: error: variable or field 'draw_bitmap' declared void

makechar.cpp:85: error: 'FT_Bitmap' was not declared in this scope

makechar.cpp:85: error: 'bitmap' was not declared in this scope

makechar.cpp:85: error: 'FT_Int' was not declared in this scope

make[1]: *** [makechar.o] エラー 1

-------------------------------------------------

対象方法として、以下を実施する。

ln -s /usr/local/include/freetype2/freetype /usr/include/freetype

/usr/local/include/freetype2/freetypeフォルダがない場合、

#yum -y install freetype freetype-devel

を実行してから、「ln -s 」のリンクコマンドを実施する。

（１）写真（レシート）

（２）識別した文字（あんまり識別度が低いようです）

ぽ

>ト

イオンM店

TEl Ti-q-8000 FAX.-q-8787

http://wu. aeon. info/

■

2 X 2g8

翻

囲

■

イオ.瓶茸牡

TV莱焼き7‐モンド 5gB

TVll'7+

2X 2g

Tvお莱缶

3X gg

マルキヨ製菓生菓千 2g7

口ール巻ポリ蟻

B Pティッシユぺーパー j78

.￣許~‐_‐一一‐‐‐‐‐‐‐‐‐‐‐‐ーー‐.‐

WrWi支払

お釣

\T.2gB

\l.2g8

おエ上げ商品撒: vo

{消費税等)

レう゛T02 20js/ol〃5(火曜日ij7:og

取瑚Tifl 担当ij■■ 藁子

T.‐‐.一‐‐.‐-.…‐‐‐‐.リu‐.一ー‐‐

=**********#*f['8

WiWi支払輯:

WrWi残輯 =

\T.2gB

判j7

(内ポイント封韓金輯 \T.2gB )

今回ポイント:

f2 P

【内ポイント2倍封寓 j2 P 〕

累計ポィント:

j42 p

■

..............................

よろず

2015年7月14日火曜日

Apache Sparkの初回試し（失敗→解決→成功→動作確認）

2015年7月10日金曜日

写真中の文字を識別するオープンソースNHOCRの使い方

NHocr: OCR engine for Japanese language

国際交流を促進するグローバル音楽フェスティバルについて語ろう。