Is it advisable to do measurement in jupyter notebook?

jupyter

#1

As it has already been discussed in this forum, jupyter notebooks are great for publishing as well as note (lab) keeping. Can you directly record and visualize data in jupyter notebook? If yes, does it have any effect on the processing speed of your measurement?


#2

Excellent questions, with no single answer. First, I guess it depends a lot on what you are used to. If you are already using Jupyter for the data analysis, it would make sense to also use it for acquiring. However, if you are using an already existent program for the acquisition, you would need to adapt it to work with the notebook. So, it is more of a balance between the effort you are willing to put, and the context.

Regarding speed, the honest answer is that it depends. Jupyter is just a layer on top of the Python kernel, which doesn’t add overhead. Moreover, data acquisition tasks tend not to be very computer intensive, because they actually run on separated devices. If you, however, do any kind of analysis while acquiring (control loops, data optimization, etc.) the situation changes. In principle Jupyter handles threads as any other Python script. This means that you could trigger a measurement inside a thread and within one of the cells, while you use another cell to update the results (in case you need to monitor the signal).

If you don’t need to monitor the signal, things are easier, since you don’t need any threads. You can wait for the acquisition to be over and then plot the results.

What I really like about using Jupyter notebooks is that they are very easy to share and you can have everything in the same place. You can keep the parameters of your measurement, you can keep the plot and you can make notes. The only drawback is that you can easily erase something, for example, you do a measurement with some parameters, you modify them and repeat. You have to organize your workflow in such a way that you don’t lose information.


#3

Indeed, jupyter notebooks are very handy for reporting and documentation. From my experience of analysis and simulations in jupyter, you have to keep track of all the variables and memory usage. Otherwise the notebook immediately consumes a lot of memory. The funny thing is even in a “for-lop” where the objects/variables should be replaced by the new ones, the memory usage gradually increases. I had to put the codes in a script and run them from command prompt.
As you mentioned data-acquisition shouldn’t make a difference.


#4

I use jupyter notebooks for data aquisistion, mainly because of the flexibility of combining multiple scans etc. I started using it when I was using hardware that takes much time to initialize/connect.

The main drawback is that “rerunning” the same commands in the same order as the “day before” can be hard, because the notebook cells can be executed in any order.

The other drawback might be memory leakage. For this implementing Queue’s instead of lists might help. Furthermore running often:

import gc
gc.collect()

might help to free memory.


#5

Indeed, the main critics of notebooks focus on the ‘randomness’ at which each cell is executed, and therefore you can’t guarantee a given state when you run a given piece of code. And this is general, not specific to lab environments.

However, you can also assume that in the same way you keep track of how you prepared a sample (for instance, by writing on a piece of paper), you can keep track of how you performed an experiment.

I have been using notebooks a bit more since my last message. I do like the fact that if you share them with someone, it is pretty clear what has happened, where the plots come from, etc. Takes a bit of establishing common rules among collaborators in order to have a good workflow though.