python - Efficiently joining two dataframes based on multiple levels of a multiindex -

-

i have dataframe large multiindex, , secondary dataframe multiindex subset of larger one. secondary dataframe kind of lookup table. want add columns lookup table larger dataframe. primary dataframe large, want efficiently.

here imaginary example, want join df2 df1:

   in [11]: arrays = [    ['sun', 'sun', 'sun', 'moon', 'moon', 'moon', 'moon', 'moon'],    ....:               ['summer', 'winter', 'winter', 'summer', 'summer', 'summer', 'winter', 'winter'],    ....:               ['one', 'one', 'two', 'one', 'two', 'three', 'one', 'two']]  in [12]: tuples = list(zip(*arrays))  in [13]: index = pd.multiindex.from_tuples(tuples, names=['body', 'season','item'])  in [14]: df1 = pd.dataframe(np.random.randn(8,2), index=index,columns=['a','b'])  in [15]: df1 out[15]:                                   b body season item sun  summer 1   -0.121588  0.272774      winter 1    0.233562 -2.005623             2   -1.034642  0.315065 moon summer 1    0.184548  0.820873             2    0.838290  0.495047             3  0.450813 -2.040089      winter 1   -1.149993 -0.498148             2    2.406824 -2.031849  [8 rows x 2 columns]   in [16]: index2= pd.multiindex.from_tuples([('sun','summer'),('sun','winter'),('moon','summer'),('moon','winter')],names=['body','season'])  in [17]: df2 = pd.dataframe(['good','bad','ugly','confused'],index=index2,columns = ['mood'])  in [18]: df2 out[18]:                  mood body season sun  summer           winter       bad moon summer      ugly      winter  confused  [4 rows x 1 columns]

now, suppose want add columns df2 df1? line way find job:

in [19]: df1 = df1.reset_index().join(df2,on=['body','season']).set_index(df1.index.names)  in [20]: df1 out[20]:                                   b      mood body season item sun  summer 1   -0.121588  0.272774           winter 1    0.233562 -2.005623       bad             2   -1.034642  0.315065       bad moon summer 1    0.184548  0.820873      ugly             2    0.838290  0.495047      ugly             3  0.450813 -2.040089      ugly      winter 1   -1.149993 -0.498148  confused             2    2.406824 -2.031849  confused  [8 rows x 3 columns]

it works, there 2 problems method. first, line ugly. needing reset index, recreate multiindex, makes simple operation seem needlessly complicated. second, if understand correctly, every time run reset_index() , set_index(), copy of dataframe created. working large dataframes, , seems inefficient.

is there better way this?

this not implemented internally atm, soln recommended one, see here issue

you can wrap in function if want make nicer. reset_index/set_index copy (though can pass inplace=true argument if want); inplace these changing index attribute.

you patch in nice function like:

def merge_multi(self, df, on):     return self.reset_index().join(df,on=on).set_index(self.index.names) dataframe.merge_multi = merge_multi  df1.merge_multi(df2,on=['body','season'])

however, merging definition creates new data, not sure how save you.

a better method build smaller frames, larger merge. might want this


Comments

Post a Comment

Popular posts from this blog

c++ - OpenCV Error: Assertion failed <scn == 3 ::scn == 4> in unknown function, -

-
i read alot other solution still confused should mine... #include <opencv2/objdetect/objdetect.hpp> #include <opencv2/highgui/highgui.hpp> #include <opencv2/imgproc/imgproc.hpp> #include <iostream> #include <stdio.h> using namespace std; using namespace cv; int main(int argc, const char** argv) { //create cascade classifier object used face detection cascadeclassifier face_cascade; //use haarcascade_frontalface_alt.xml library face_cascade.load("haarcascade_frontalface_alt.xml"); //setup video capture device , link first capture device videocapture capturedevice; capturedevice.open(0); //setup image files used in capture process mat captureframe; mat grayscaleframe; //create window present results namedwindow("outputcapture", 1); //create loop capture , find faces while (true) { //capture new image frame capturedevice >> captureframe; ...
Read more

php - render data via PDO::FETCH_FUNC vs loop -

-
i've been using 2 methods render data. the first one: function name($id,$name){ return '<div id="'.$id.'">'.$name.'</div>'; } echo implode($pdo->query("select id,name user")->fetchall(pdo::fetch_func,'name')); the second one: $users = $pdo->query("select id,name user")->fetchall(pdo::fetch_obj); foreach($users $user){ echo name($user->id,$user->name); } i don't understand how pdo::fetch_func works. tried figure out. however, not well-documented. could please explain fetch mode? , also, 1 performs better? thank you. both methods wrong , have learn how use templates , how separate business logic presentation logic. $users = $pdo->query("select id,name user")->fetchall(pdo::fetch_obj); tpl::assign('users', $users); is code business logic part. then in template <?php foreach $users $row): ?> <div id="<?=$...
Read more

The canvas has been tainted by cross-origin data in chrome only -

-
uncaught securityerror: failed execute 'getimagedata' on 'canvasrenderingcontext2d': canvas has been tainted cross-origin data i getting above error in chrome. code(following) works fine in mozilla. have not corss domain issue. why above error in chrome? var wheel_canvas, wheel_context, can_w, can_h, wheel_grd; var large_radius = 160; var small_radius = 120; var wheel_canvas = document.getelementbyid('wheel_canvas'); var wheel_context = wheel_canvas.getcontext('2d'); var can_w = $(wheel_canvas).width(); var can_h = $(wheel_canvas).height(); var centerx = can_w / 2, centery = can_h / 2; var wheel_grd = null; test('#ff0000'); function test(hex) { $('#clrpicker').css({'left': 210, 'top': 210 }); inverted = hexbw(hex); $('#color_val_show').val(hex); current_color_hex_val_selected = hex; $('#color_val_show').css({'color':inverted,'background':hex}); fillgradient...
Read more