scala - How can I load Avros in Spark using the schema on-board the Avro file(s)? -

-

i running cdh 4.4 spark 0.9.0 cloudera parcel.

i have bunch of avro files created via pig's avrostorage udf. want load these files in spark, using generic record or schema onboard avro files. far i've tried this:

import org.apache.avro.mapred.avrokey import org.apache.avro.mapreduce.avrokeyinputformat import org.apache.hadoop.io.nullwritable import org.apache.commons.lang.stringescapeutils.escapecsv  import org.apache.hadoop.fs.path import org.apache.hadoop.fs.filesystem import org.apache.hadoop.conf.configuration import java.net.uri import java.io.bufferedinputstream import java.io.file import org.apache.avro.generic.{genericdatumreader, genericrecord} import org.apache.avro.specific.specificdatumreader import org.apache.avro.file.datafilestream import org.apache.avro.io.datumreader import org.apache.avro.file.datafilereader import org.apache.avro.mapred.fsinput  val input = "hdfs://hivecluster2/securityx/web_proxy_mef/2014/05/29/22/part-m-00016.avro" val inuri = new uri(input) val inpath = new path(inuri)  val fsinput = new fsinput(inpath, sc.hadoopconfiguration) val reader =  new genericdatumreader[genericrecord] val datafilereader = datafilereader.openreader(fsinput, reader) val schemastring = datafilereader.getschema  val buf = scala.collection.mutable.listbuffer.empty[genericrecord] while(datafilereader.hasnext)  {   buf += datafilereader.next } sc.parallelize(buf)

this works 1 file, can't scale - loading data local ram , distributing across spark nodes there.

to answer own question:

import org.apache.spark.sparkcontext import org.apache.spark.sparkcontext._  import org.apache.avro.generic.genericrecord import org.apache.avro.mapred.avrokey import org.apache.avro.mapred.avroinputformat import org.apache.avro.mapreduce.avrokeyinputformat import org.apache.hadoop.io.nullwritable import org.apache.commons.lang.stringescapeutils.escapecsv  import org.apache.hadoop.fs.filesystem import org.apache.hadoop.fs.path import org.apache.hadoop.conf.configuration import java.io.bufferedinputstream import org.apache.avro.file.datafilestream import org.apache.avro.io.datumreader import org.apache.avro.file.datafilereader import org.apache.avro.file.datafilereader import org.apache.avro.generic.{genericdatumreader, genericrecord} import org.apache.avro.mapred.fsinput import org.apache.avro.schema import org.apache.avro.schema.parser import org.apache.hadoop.mapred.jobconf import java.io.file import java.net.uri  // spark-shell -usejavacp -classpath "*.jar"  val input = "hdfs://hivecluster2/securityx/web_proxy_mef/2014/05/29/22/part-m-00016.avro"  val jobconf= new jobconf(sc.hadoopconfiguration) val rdd = sc.hadoopfile(   input,   classof[org.apache.avro.mapred.avroinputformat[genericrecord]],   classof[org.apache.avro.mapred.avrowrapper[genericrecord]],   classof[org.apache.hadoop.io.nullwritable],   10) val f1 = rdd.first val = f1._1.datum a.get("rawlog") // access avro fields

Comments

Post a Comment

Popular posts from this blog

c++ - OpenCV Error: Assertion failed <scn == 3 ::scn == 4> in unknown function, -

-
i read alot other solution still confused should mine... #include <opencv2/objdetect/objdetect.hpp> #include <opencv2/highgui/highgui.hpp> #include <opencv2/imgproc/imgproc.hpp> #include <iostream> #include <stdio.h> using namespace std; using namespace cv; int main(int argc, const char** argv) { //create cascade classifier object used face detection cascadeclassifier face_cascade; //use haarcascade_frontalface_alt.xml library face_cascade.load("haarcascade_frontalface_alt.xml"); //setup video capture device , link first capture device videocapture capturedevice; capturedevice.open(0); //setup image files used in capture process mat captureframe; mat grayscaleframe; //create window present results namedwindow("outputcapture", 1); //create loop capture , find faces while (true) { //capture new image frame capturedevice >> captureframe; ...
Read more

php - render data via PDO::FETCH_FUNC vs loop -

-
i've been using 2 methods render data. the first one: function name($id,$name){ return '<div id="'.$id.'">'.$name.'</div>'; } echo implode($pdo->query("select id,name user")->fetchall(pdo::fetch_func,'name')); the second one: $users = $pdo->query("select id,name user")->fetchall(pdo::fetch_obj); foreach($users $user){ echo name($user->id,$user->name); } i don't understand how pdo::fetch_func works. tried figure out. however, not well-documented. could please explain fetch mode? , also, 1 performs better? thank you. both methods wrong , have learn how use templates , how separate business logic presentation logic. $users = $pdo->query("select id,name user")->fetchall(pdo::fetch_obj); tpl::assign('users', $users); is code business logic part. then in template <?php foreach $users $row): ?> <div id="<?=$...
Read more

The canvas has been tainted by cross-origin data in chrome only -

-
uncaught securityerror: failed execute 'getimagedata' on 'canvasrenderingcontext2d': canvas has been tainted cross-origin data i getting above error in chrome. code(following) works fine in mozilla. have not corss domain issue. why above error in chrome? var wheel_canvas, wheel_context, can_w, can_h, wheel_grd; var large_radius = 160; var small_radius = 120; var wheel_canvas = document.getelementbyid('wheel_canvas'); var wheel_context = wheel_canvas.getcontext('2d'); var can_w = $(wheel_canvas).width(); var can_h = $(wheel_canvas).height(); var centerx = can_w / 2, centery = can_h / 2; var wheel_grd = null; test('#ff0000'); function test(hex) { $('#clrpicker').css({'left': 210, 'top': 210 }); inverted = hexbw(hex); $('#color_val_show').val(hex); current_color_hex_val_selected = hex; $('#color_val_show').css({'color':inverted,'background':hex}); fillgradient...
Read more