Monday, 15 September 2014

scala - Cannot compile Spark Streaming example: getting updateStateByKey is not a member of org.apache.spark.streaming.dstream.DStream error -


Text after "div class =" itemprop = "text">

I want to understand better streaming of SPARC so I copied the StatefulNetworkWordCount.scala example in my own directory, and pasted it As is , a simple SBT config file has been created, and tried to compile but it does not seem to work. Why is the complaint about the API? Any remedy? The original example just compiles, but I want to change it without compiling it to the whole spark directory, as if it is set by default.

Thank you! Matte

  $ more name build.sbt: = "simple project" version: = "1.0" scalaVersion: = "2.10.4" libraryDependencies + = "org.apache.spark" %% " Spark-Core "%" 1.2.1 "Library Developments +" org.apache.spark "%" Spark-Streaming_2.10 "%" 1.2.1 "Urban Lok-2: Streaming Test Melliner $   
  urbanlegends -2: streamingtest mlieber $ sbt package [info] simple project current project set (in construction file: / users / mlieber / app / spark2 / spark / Example / lieber / streamingtest /) [info] Compilation 2 To make sources for /Users/mlieber/app/spark2/spark/examples/lieber/streamingtest/target/scala-2.10/classes ... [Error] / users / mlieber / apps / spark2 / spark / examples / lieber / Streamingtest / src / main / scala / com / cloudera / streamingtest / StatefulNetworkWordCount2.scala: 58: value updateStateByKey org.apache.spark.streaming.dstream.DStream [[Not a member of string, int]] [Error] ** Val State Dustum = Word Dustum. [Update] [Entity] (NewUpdatefunk, [Error] ** ^ [Error] Received an Error [Error] (Compilation: Compilation) Compilation Failed [Error] Total Time: 3S, Complete 16 March 2 015 4:23:15 PM Urban Public-2: Streaming Test Meliverver $  

The problem seems to be incompatibility of libraries with "Spark-Core"% "1.2.1" and "Spark-Streaming_2.10"% "1.2.1" Try changing

.  "Spark-Core"% "1.2.0" "Spark-Streaming_2.10"% "1.2.0"  

[Update] Or You also need to add import org.apache.spark.streaming.StreamingContext ._


No comments:

Post a Comment