Serial Data Logger

with PIC microcontroller

      Main           Schematics           Software           Construction      Other Projects  








Because of the distributed nature of this project, there are several pieces of software you will need. You can get all of those from the code table on the right. The following is a description of the software you will need. The text below may contain a bit more information than strictly necessary - you should be able to use everything without knowing anything anything about the way checksumming works or how the noise reduction algorithm operate, but I'm including this too, just in case you want to do some customization. If you need additional information, please email me - my address is at the bottom of each page.


Data capture and serialization
The first piece of code is the program for the microcontroller (logger.asm). It is a very short assembly program that simply reads the microcontroller's analog input and sends out the same data in serial format.

Because the ADC operates with a 10-bit precision, each data sample point is serialized as two bytes (i.e. 16 bits). When the data is picked up on the other side of the serial connection, the receiver must figure out which is the high and which is the low byte. In order to help with this and to ensure some level of error correction, the microcontroller allocates 6 unused bytes for checksum, as shown below:

Checksum algorithm


Each of the five checksum bytes are calculated as inverse-XOR of two data bits, as shown on the diagram. The last bit of the high-order byte is always set to 0. This encoding allows the receiver to synchronize on the data stream and provides some level of error checking.


Storing the serialized data
I wrote a simple Perl script for the purpose. The script reads the data from the serial port and records it in a file.

Usage

perl logger [-PARAMETER value]

ParameterDefaultDescription
INPUTin.dat Name of the input data file.
-s10 Scope rate. The "scope" is the graph of the data coming from the logger displayed on the screen while the program is running. A value of "10" for example means that each 10-th sample from the logger will be drawn.
-f4096 Frame size. Refer to the file format diagram below for more information.
-lSTDOUT Name of the output file.
-p/dev/ttyS0 Serial port.


The data logging script is designed for a Linux (UNIX) computer - in case you want to run the data logger on a different OS, you might have to either adjust the script or write your custom program. If you do so, it might help you to store the data in the same format as this script does - this will allow the OS-independent data processing and visualisation scripts (described below) work with your file.

The data is stored in the file by frames. Frame size is variable. Each frame is timestamped - this allows you, from the timestamps of two subsequent frames, to identify the exact time each sample was recorded. The format of the file is shown in the next diagram:

Logfile format


Timestamps and frame sizes are stored MSB first. Data inside each frame is stored byte-by-byte, as it comes from the logger via the serial interface. Because each sample consists of two bytes, and considering that occasionally a byte or two can be lost in transmission, it is obvious that a synchronization mechanism for interpreting the data in the frames is needed. This is achieved via the checksumming mechanism described in the previous section. If the checksum of two bytes do not match, the sample is considered bad data and an attempt to read the sequence from the next byte is made. This is repeated until the checksum matches, at which point synchronization has been achieved.

The datlib.pl library can be used for reading data in this format.


Processing and visualisation of stored data
So, you have your data logging system running, happily storing file after file of high-quality data. But what use is the data if you can't process and visualise it? I have some libraries and scripts that will let you do some basic editing and visualisation of your data.

The clip script allows you to select part of your data. For example, if you recorded 24 hours of data in a single file, you may want to split this into 24 one-hour long pieces and work on these separately. The options of the clip script are described below:

Usage

perl clip [-PARAMETER value] ...

ParameterDefaultDescription
INPUTin.dat Name of the input data file.
OUTPUTout.dat Name of the output data file.
STARTnone Start timesatamp in YYYYMMDDHHMMSS format. If specified, all frames before the given timestamp will be excluded from the selection.
ENDnone End timesatamp in YYYYMMDDHHMMSS format. If specified, all frames after the given timestamp will be excluded from the selection.
DEBUG1 Show debug information while running the script. If you want to turn debugging off, set this to 0.


After you've extracted the piece of data you need, you will probably want to visualise the data on a graph. To plot your data, use the draw script:

Usage

perl draw [-PARAMETER value] ...

ParameterDefaultDescription
INPUTin.dat Name of the input data file.
OUTPUTout.png Name of the output image file. Only PNG format is supported.
INVERT0 If set to 1 the Y-axis of the graph will be inverted. Default is 0.
IMG_WIDTH800 Width of the output image, in pixels.
IMG_HEIGHT500 Height of the output image, in pixels.
BORDER10 Width of the border around the graph, in pixels.
BACKGROUND_COLOR#000000 Image background color, hex string.
BORDER_COLOR#FFFFFF Color of the graph border, hex string.
DATA_COLOR#FF0000 Color of the graph data line, hex string.
TIMEMARK_COLOR#808080 Color of the time marks, hex string.
TIMEMARK_INTERVAL3600 Interval between timemarks displayed on the diagram, in seconds.
TIMEMARK_LABEL_RATIO3 Number of marks between timemark labels. For example, 3 means that each third timemark will be labeled.
TIMEMARK_LABEL_OFFSET5 Distance between each timemark and its label, in pixels.
LABEL_OFFSET10 Distance between labels and the diagram, in pixels.
LABELnone Diagram label. Use quotes when specifying multiword labels.
SMOOTHING_FILTER_SIZE1 Controls the data pre-filter. Number of samples to be averaging for the pre-filter. To enable averaging, set to a value larger than 1. See Filtering and Jump Correction for more information.
JUMP_CORRECTION0 Enable or disable the jump correction. Set to 1 to enable. See Filtering and Jump Correction for more information.
JC_T12.0 T1 time constant for the jump correction, in seconds. See Filtering and Jump Correction for more information.
JC_T20.1 T2 time constant for the jump correction, in seconds. See Filtering and Jump Correction for more information.
JC_THRESHOLD1.0 Jump correction threshold. See Filtering and Jump Correction for more information.
POST_FILTER0 Enable/disable post-filter. Set to 1 to enable. See Filtering and Jump Correction for more information.
PF_T10 Time constant for the post-filter, in seconds. See Filtering and Jump Correction for more information.
YSCALE1.0 Controls the scale along the vertical axis.
DEBUG1 Controls the display of debugging information when the script is running. The default is 1 (debug messages displayed). Set to 0 to disable debugging.



Filtering and Jump correction
While using the data logger in one of my projects - a sensitive magnetometer - I had to find a solution of a very annoying problem. The magnetometer I built was for recording magnetic storms. Those are slow changes in Earth's magnetic field and can last for days. During this time, the magnetometer have to deal with a number of disturbances from local sources. For example, each time the neighbourhood kids parked their bicycle near our patio, the readings jumped this or that direction. Same thing happened when somebody opened the door of the refrigerator or pushed the outdoor furniture around. All these dirsturbances were presenting a very noisy environment in which the slight changes in the Earth magnetic field were lost.

Fortunately, there was an easy way to solve those problems, without banning everybody from moving, or having to relocate to the middle of the Mojave desert. Most of those disturbances differ from the changes in planetary magnetic field by their speed. Somebody pushes a iron chair around - this is detected as a very fast shift in the direction of the magnetic field that takes place in the duration of a second or two, or a jump in the magnetometer readings, like this:

Uncorrected jump in readings


I added something I called "jump correction" to the data processing library. If jump correction is turned on, any sudden jumps are corrected for:

Readings after applying jump correction


The precise speeds and thresholds for activation of the jump correction algorithm are controlled by the respective parameters, listed in the usage info for the draw script.



Software - Serial Data Loggerv.1.0, 2011/03/01
PIC16F688 program - assembly codelogger.asm
PIC16F88 program - precompiled HEX file logger.HEX
Perl scripts for data logging and processingscripts.zip







      Main           Schematics           Software           Construction      Other Projects  
 
my email address