Using the Serial Flash for Storage

Note: This example builds on the previous example titled: Adding a Timer to Schedule Readings. Only the sections of code which have been changed or added are discussed here. However, a full copy of the complete sketch can be found at the end of this page.

In this example we will demonstrate using the Serial Flash storage on the SODAQ Mbili to store the data which has been read from the sensors.  This example builds on the previousAdding a Timer to Schedule Reading example, but instead of using the MicroSD device for storage it uses the Serial Flash storage.

Theory

The SPI Protocol

The Serial Peripheral Interface (SPI) is a synchronous serial data protocol used by the SODAQ Mbili for communication with the MicroSD device and the Serial Flash. On other Arduino microcontrollers it is used for communication with a variety of peripherals and can also be used for communication between two microcontrollers.

AT45DB161E Serial Flash

The SODAQ Mbili includes a AT45DB161E Serial Flash chip which is capable of storing 16mbit of data and is connected via the Serial Peripheral Inteface (SPI),

Unlike with the MicroSD storage, the Serial Flash does not have a file system but is essentially a large linear storage buffer. This storage buffer is divided into pages each storing a fixed amount of data. In the case of the AT45DB161E there are 4096 pages each capable of storing 528 bytes.

The Serial Flash storage is not accessed directly but through one of the two available SRAM buffers. To read data from the Serial Flash, first the specific page on which that data exists must be copied onto one of the SRAM buffers. Then that data can be access via that SRAM buffer. To write data to the Serial Flash, the data must be first copied onto one of the SRAM buffers and then that buffer is written to a specific page.

Sodaq_dataflash Library

The Sodaq_dataflash library contains the class Sodaq_dataflash which provides the functionality for working with the Serial Flash storage. The library also declares a global object of type Sodaq_Dataflash named dflash.

Note: The AT45DB161E has two available SRAM buffers, however, the current implementation of the Sodaq_dataflash library only uses the first buffer.

Additional Required Components

  • None

Additional Required Libraries

  • Sodaq_dataflash

Library Installation

The Sodaq_dataflash library is included with the SODAQ Mbili files that you have already installed.

If necessary, refer to Section 2 of the Getting Started guide for details on where to download from and how to install the SODAQ Mbili files.

Hardware Setup

You should refer to both the board diagram and Grove sockets page for additional information.

  1. First, plug the TPH Sensor into the Grove I2C socket.
  2. Then, plug the 0.5W solar panel and the 1A LiPo battery into their respective sockets.

Turn on the SODAQ Mbili board, compile and upload the following sketch from the Arduino IDE onto the SODAQ Mbili board. Leave the USB cable plugged in and open the Serial Monitor (Ctrl-Shift-M) and ensure that it is set to the 9600 baud rate.

After opening the Serial Monitor (Ctrl-Shift-M), you should see output similar to this:

Additional Sketch Code

Libarary Includes

In addition to the existing libraries, we must now also include the Sodaq_dataflash library in the sketch using the #include compiler directive.

#include <Sodaq_dataflash.h>

Globals

Here we first define the number of memory pages (NUM_PAGES) and the size of each page (PAGE_SIZE in bytes). The values for the AT45DB161E are 4096 pages each with size of 528 bytes. Additionally, we define the character(s) which specify a new line. These are platform specific and ere we specify the newline characters for the Windows platform.

We also declare two global variables which define the current page and current position (address) on that page. These are both initialised to 0.

The line #define DEBUG specifies a preprocessor directive DEBUG. This is used throughout the sketch to switch on or off sections of code. For testing purposes, additional sections of code have been added which are used to copy the stored data from the Serial Flash onto a file stored on the MicroSD card. If this line (#define DEBUG) is commented out, all sections which fall between #ifdef DEBUG and #endif, will be ignored by the compiler.

For example: Three additional global variables have been defined here in one of these DEBUG sections of code. These are used to track the current reading number, to specify how often to copy the data to the MicroSD card, and to keep track of whether the Serial Flash buffer has wrapped to the start. We have also moved the MicroSD constants into this section as they are only used in a DEBUGВ sections of code.

//DataFlash constants
#define NUM_PAGES 4096
#define PAGE_SIZE 528
#define NEW_LINE "\r\n"

//DataFlash variables
int Page = 0; 
int Address = 0;

#define DEBUG

#ifdef DEBUG
int ReadingCount = 0;
int DumpEvery = 20;
bool HasWrapped = false;

//The data log file
#define FILE_NAME "DataLog.txt"

//Digital pin 11 is the MicroSD slave select pin on the Mbili
#define SD_SS_PIN 11
#endif

setupLogFile()

Here we initialize the data flash storage with a call to dflash.init() passing the parameters for the pins: MISO, MOSI, SCK, & SS. Additionally, if the DEBUG compiler directive has been defined, we also attempt to initialize the MicroSD device, and display an error message (in the Serial Monitor) if that fails.

void setupLogFile()
{
    //Initialize the Data Flash
    dflash.init(MISO, MOSI, SCK, SS);
 
    #ifdef  DEBUG
    //Initialize the SD Card
    if (!SD.begin(SD_CS_PIN))
    {
         Serial.println("Error: SD card failed to initialise or is missing.");
    }
#endif
}

logData()

In the existing example, the user defined method createDataRecord() generates a String which contains the readings in Comma Separated Values (CSV) in ASCII format. This String is then passed as the parameter rec to this method logData(). Arguably it would be more efficient to store the data in a binary form to reduce the storage space requirements. However, to keep things simple we will leave the createDataRecord() method unchanged and simply modify this method to now store the data on the Serial Flash.

Here we start by adding the newline character(s) to rec. Next we copy the data from rec into a buffer of type uint8_t as this is the format used by the methods in theSodaq_dataflash library.

Note: The method String.getBytes() automatically sets the last character of the buffer to be the null termination character (\0), and so it is necessary to specify a buffer which has space to store one additional character.

First we check to see if the amount of data we need to write fits in the current page. If it does, we then copy it to the SRAM buffer using dflash.writeStrBuf1() and then write that SRAM buffer to the current page using dflash.writeBuf1ToPage(). We then increment the address on the current page to equal the address of the position after the data we just wrote.

If the data does not fit on the current page, then we must take a few additional steps. First we calculate how much of the data can fit on the current page and then write that portion of the data to the current page using the steps listed above. Next, we increment the current page and by using modular division % 4096 we ensure that value of Page is always in the range of 0…4095. This causes Page to reset to 0 when it reaches the end (wraps). If DEBUG is defined and if Page has wrapped to 0, we set the boolean HasWrapped to true. We then write the remaining data to the new page using the same steps as above.

Finally, if DEBUG is defined we check to see if we are due to copy the data from the Serial Flash storage to a file on the MicroSD card. If so we call the user defined method copyToSDCard().

void logData(String rec)
{
   //Add new line characters
    rec  +=  NEW_LINE;
 
   //Copy rec to an array  
    int len = rec.length();
    uint8_t buff[len+1]; //+1 for null termination character
    rec.getBytes(buff, len+1);

   //If it fits on the current page
    if ((Address + len) < PAGE_SIZE)
    {
       dflash.writeStrBuf1(Address, buff, len);
       dflash.writeBuf1ToPage(Page);
       Address  +=  len;
    }
     else
    {
        //Work out the amount to be written to the 
        //current and next page
        int part = PAGE_SIZE - Address;
        int other = len - part;
 
        //Copy the first part to the current page
        dflash.writeStrBuf1(Address, buff, part);
        dflash.writeBuf1ToPage(Page);
 
        //Increment the page number and wrap if needed
        Page++;
        Page = Page % NUM_PAGES;

#ifdef DEBUG
         if (Page == 0)
        {
            HasWrapped  =  true;
        }
#endif 
  
        //Copy the remainder to the next page
        dflash.writeStrBuf1(0,В &buff[part],В other);
        dflash.writeBuf1ToPage(Page);
        Address = other;
    }
 
//If in debug mode check to see if we need to dump to SD card
#ifdef DEBUG
    if ((ReadingCount != 0) && (ReadingCount % DumpEvery) == 0)
    {
       copyToSDCard();
    }
 
    ReadingCount++;
#endif
}

copyToSDCard()

This method is used to copy the data from the Serial Flash storage to a file (DataLog.txt) on a MicroSD card. This code is enclosed within the DEBUG compiler directive and is only included in the compiled sketch if DEBUG has been defined.

First we attempt to delete the log file if one already exists. We then create a new file and write the data header to the first line. Next we copy each of the full pages from the Serial Flash storage to the file stored on the MicroSD card. There are two scenarios here. If the Serial Flash storage has wrapped, we start by copying all the full pages which have an index higher than the current page. Then we copy the full pages that have an index lower than the current page. If the we have not wrapped we do not need to copy the full pages with indexes higher than the current one. We then copy the data from the current page, the amount of which depends on how much of the current page we have filled so far (specified by the data before Address). Finally we close the file which saves it to the MicroSD card.

If you remove the MicroSD card and insert it into your PC you should find a file (DataLog.txt) in the root folder which contains the same data as was shown in the Serial Monitor.

#ifdef DEBUG
void copyToSDCard()
{
    Serial.println("Copying data from Serial Flash to SD card");
 
    //Erase existing log file
    SD.remove(FILE_NAME);
 
    //Create new file and write header
    File logFile = SD.open(FILE_NAME, FILE_WRITE);
    logFile.println(DATA_HEADER);

    //Copy Buffer
    uint8_t buff[PAGE_SIZE];

    //If we have already wrapped around
    //Copy all pages in front of current page
     if (HasWrapped)
    {
      for (int i=(Page+1); i<NUM_PAGES; i++)
        {
            dflash.readPageToBuf1(i);
            dflash.readStrBuf1(0,В buff,В PAGE_SIZE);
            logFile.write(buff, PAGE_SIZE);
        }
   }
  
    //Copy over each proceeding full page
    for (int i=0; i<Page; i++)
    {
        dflash.readPageToBuf1(i);
        dflash.readStrBuf1(0,В buff,В PAGE_SIZE);
        logFile.write(buff, PAGE_SIZE);
   }
  
     //Copy over the partial current page
    dflash.readPageToBuf1(Page);
    dflash.readStrBuf1(0,В buff,В Address);
    logFile.write(buff, Address);
  
    //Close the file to save it
    logFile.close();
}
#endif

Full Sketch Code

#include <Wire.h>
#include <SPI.h>
#include <SD.h>

//SODAQ Mbili libraries
#include <RTCTimer.h>
#include <Sodaq_BMP085.h>
#include <Sodaq_SHT2x.h>
#include <Sodaq_DS3231.h>
#include <Sodaq_dataflash.h>

//DataFlash constants
#define NUM_PAGES 4096
#define PAGE_SIZE 528
#define NEW_LINE "\r\n"

//The delay between the sensor readings 
#define READ_DELAY 1000

//Data header
#define DATA_HEADER "TimeDate, TempSHT21, TempBMP, PressureBMP, HumiditySHT21"

//TPH BMP sensor
Sodaq_BMP085 bmp;

//RTC Timer
RTCTimer timer;

//DataFlash variables
int Page = 0; 
int Address = 0;

#define DEBUG

#ifdef DEBUG
int ReadingCount = 0;
int DumpEvery = 20;
bool HasWrapped = false;

//The data log file
#define FILE_NAME "DataLog.txt"

//Digital pin 11 is the MicroSD slave select pin on the Mbili
#define SD_SS_PIN 11
#endif

void setup() 
{
    //Initialise the serial connection
    Serial.begin(9600);
    
    //Initialise sensors
    setupSensors();
        
    //Initialise log file
    setupLogFile();
    
    //Setup timer events
    setupTimer();
    
    //Echo the data header to the serial connection
    Serial.println(DATA_HEADER);

    //Take first reading immediately
    takeReading(getNow());
}

void loop() 
{
    //Update the timer 
    timer.update();    
}

void takeReading(uint32_t ts)
{
    //Create the data record
    String dataRec = createDataRecord();
    
    //Save the data record to the log file
    logData(dataRec);
    
    //Echo the data to the serial connection
    Serial.println(dataRec);
}

void setupSensors()
{
    //Initialise the wire protocol for the TPH sensors
    Wire.begin();
    
    //Initialise the TPH BMP sensor
    bmp.begin();

    //Initialise the DS3231 RTC
    rtc.begin();
}

void setupLogFile()
{
    //Initialise the Data Flash
    dflash.init(MISO, MOSI, SCK, SS);
    
#ifdefВ DEBUG
    //Initialise the SD Card
    if (!SD.begin(SD_SS_PIN))
    {
        Serial.println("Error: SD card failed to initialise or is missing.");
    }
#endif
}

void setupTimer()
{
    //Instruct the RTCTimer how to get the current time reading
    timer.setNowCallback(getNow);

    //Schedule the reading every second
    timer.every(READ_DELAY,В takeReading);
}

void logData(String rec)
{
    //Add new line characters
    recВ +=В NEW_LINE;
    
    //Copy rec to an array  
    int len = rec.length();
    uint8_tВ buff[len+1];В //+1 for null termination character
    rec.getBytes(buff, len+1);
    
    //If it fits on the current page
    if ((Address + len) < PAGE_SIZE)
    {
        dflash.writeStrBuf1(Address,В buff,В len);
        dflash.writeBuf1ToPage(Page);
        AddressВ +=В len;
    }
    else
    {
        //Work out the amount to be written to the 
        //current and next page
        int part = PAGE_SIZE - Address;
        int other = len - part;
        
        //Copy the first part to the current page
        dflash.writeStrBuf1(Address,В buff,В part);
        dflash.writeBuf1ToPage(Page);
        
        //Increment the page number and wrap if needed
        Page++;
        PageВ =В PageВ %В NUM_PAGES;

#ifdef DEBUG
        if (Page == 0)
        {
            HasWrappedВ =В true;
        }
#endif
        
        //Copy the remainder to the next page
        dflash.writeStrBuf1(0,В &buff[part],В other);
        dflash.writeBuf1ToPage(Page);
        AddressВ =В other;
    }
    
//IfВ inВ debugВ modeВ checkВ toВ seeВ ifВ weВ needВ toВ dumpВ toВ SDВ card
#ifdefВ DEBUG
    if ((ReadingCount != 0) && (ReadingCount % DumpEvery) == 0)
    {
        copyToSDCard();
    }
    
    ReadingCount++;
#endif
}

#ifdefВ DEBUG
void copyToSDCard()
{
    Serial.println("Copying data from Serial Flash to SD card");
            
    //Erase existing log file
    SD.remove(FILE_NAME);
            
    //Create new file and write header
    File logFile = SD.open(FILE_NAME, FILE_WRITE);
    logFile.println(DATA_HEADER);

    //Copy Buffer
    uint8_tВ buff[PAGE_SIZE];

    //If we have already wrapped around
    //Copy all pages in front of current page
    if (HasWrapped)
    {
        for (int i=(Page+1); i<NUM_PAGES; i++)
        {
            dflash.readPageToBuf1(i);
            dflash.readStrBuf1(0,В buff,В PAGE_SIZE);
            logFile.write(buff, PAGE_SIZE);
        }
    }
    
    //Copy over each proceeding full page
    for (int i=0; i<Page; i++)
    {
        dflash.readPageToBuf1(i);
        dflash.readStrBuf1(0,В buff,В PAGE_SIZE);
        logFile.write(buff, PAGE_SIZE);
    }
    
    //Copy over the partial current page
    dflash.readPageToBuf1(Page);
    dflash.readStrBuf1(0,В buff,В Address);
    logFile.write(buff, Address);
    
    //Close the file to save it
    logFile.close();
}
#endif

String createDataRecord()
{
    //Create a String type data record in csv format
    //TimeDate, TempSHT21, TempBMP, PressureBMP, HumiditySHT21
    String data = getDateTime() + ", ";
    dataВ +=В String(SHT2x.GetTemperature())  + ", ";
    dataВ +=В String(bmp.readTemperature()) + ", ";
    dataВ +=В String(bmp.readPressure() / 100)  + ", ";
    dataВ +=В String(SHT2x.GetHumidity());
    
    return data;
}

String getDateTime()
{
    String dateTimeStr;
    
    //Create a DateTime object from the current time
    DateTimeВ dt(rtc.makeDateTime(rtc.now().getEpoch()));

    //Convert it to a String
    dt.addToString(dateTimeStr);
    
    return dateTimeStr;  
}

uint32_tВ getNow()
{
    return millis();
}