python - Parallel code hangs if run first, but works if run after running non-parallel code -

-

so sounds bit convoluted, i've had problem joblib lately create bunch of processes , hang there (aka, each process takes memory, uses no cpu time).

here simplest code i've got reproduce problem:

from sklearn import linear_model import numpy np sklearn import cross_validation cval joblib import parallel, delayed  def fit_hanging_model(n=10000, nx=10, ny=32, ndelay=10,                        n_cvs=5, n_jobs=none):     # create data     x = np.random.randn(n, ny*ndelay)     y = np.random.randn(n, nx)      # create model + cv     model = linear_model.ridge(alpha=1000.)     cvest = cval.kfold(n, n_folds=n_cvs, shuffle=true)      # fit model     par = parallel(n_jobs=n_jobs, verbose=10)     parfunc = delayed(_fit_model_cvs)     par(parfunc(x, y, train, test, model)                       i, (train, test) in enumerate(cvest))  def _fit_model_cvs(x, y, train, test, model):     model.fit(x, y)    n = 10 = np.random.randn(n, 32)  b = np.random.randn(32, 10)  ########## c = np.dot(a, b) ##########  fit_hanging_model(n_jobs=3)

here happens:

  • if run of code above, spawns off 3 processes , hangs
  • if run of code above, use n_jobs=1, works fine
  • if run of code above second time, after running once n_jobs=1, works fine no matter how many jobs use.
  • if run of code above except code between ######, runs fine.
  • however, if run code between ######, , try run fit_hanging_model n_jobs > 1, hangs

this joblib = 0.8.0, , sklearn 0.15-git.

note, bug on centos on linux. have not been able reproduce bug on machine, may hard reproduce.

does have idea why might going on? seems dot product doing strange, have no idea be...i'm @ end of rope...

figured out. apparently issue joblib creating multiple python processes, while mkl simultaneously trying threading. see issue , solution (which involves setting environment variables) here:

https://github.com/joblib/joblib/issues/138


Comments

Post a Comment

Popular posts from this blog

c++ - OpenCV Error: Assertion failed <scn == 3 ::scn == 4> in unknown function, -

-
i read alot other solution still confused should mine... #include <opencv2/objdetect/objdetect.hpp> #include <opencv2/highgui/highgui.hpp> #include <opencv2/imgproc/imgproc.hpp> #include <iostream> #include <stdio.h> using namespace std; using namespace cv; int main(int argc, const char** argv) { //create cascade classifier object used face detection cascadeclassifier face_cascade; //use haarcascade_frontalface_alt.xml library face_cascade.load("haarcascade_frontalface_alt.xml"); //setup video capture device , link first capture device videocapture capturedevice; capturedevice.open(0); //setup image files used in capture process mat captureframe; mat grayscaleframe; //create window present results namedwindow("outputcapture", 1); //create loop capture , find faces while (true) { //capture new image frame capturedevice >> captureframe; ...
Read more

php - render data via PDO::FETCH_FUNC vs loop -

-
i've been using 2 methods render data. the first one: function name($id,$name){ return '<div id="'.$id.'">'.$name.'</div>'; } echo implode($pdo->query("select id,name user")->fetchall(pdo::fetch_func,'name')); the second one: $users = $pdo->query("select id,name user")->fetchall(pdo::fetch_obj); foreach($users $user){ echo name($user->id,$user->name); } i don't understand how pdo::fetch_func works. tried figure out. however, not well-documented. could please explain fetch mode? , also, 1 performs better? thank you. both methods wrong , have learn how use templates , how separate business logic presentation logic. $users = $pdo->query("select id,name user")->fetchall(pdo::fetch_obj); tpl::assign('users', $users); is code business logic part. then in template <?php foreach $users $row): ?> <div id="<?=$...
Read more

The canvas has been tainted by cross-origin data in chrome only -

-
uncaught securityerror: failed execute 'getimagedata' on 'canvasrenderingcontext2d': canvas has been tainted cross-origin data i getting above error in chrome. code(following) works fine in mozilla. have not corss domain issue. why above error in chrome? var wheel_canvas, wheel_context, can_w, can_h, wheel_grd; var large_radius = 160; var small_radius = 120; var wheel_canvas = document.getelementbyid('wheel_canvas'); var wheel_context = wheel_canvas.getcontext('2d'); var can_w = $(wheel_canvas).width(); var can_h = $(wheel_canvas).height(); var centerx = can_w / 2, centery = can_h / 2; var wheel_grd = null; test('#ff0000'); function test(hex) { $('#clrpicker').css({'left': 210, 'top': 210 }); inverted = hexbw(hex); $('#color_val_show').val(hex); current_color_hex_val_selected = hex; $('#color_val_show').css({'color':inverted,'background':hex}); fillgradient...
Read more