Jump to content
43oh

POSIX IPC - shared memory + mmap


Recommended Posts

So, possibly everyone who reads this section of the forums knows I was working on translating a custom NEMA 2000 CANBus protocol. Well, I still am, but got a bit sidetracked after attempting to use an http + websocket library . . . Short story of why follows . . . Aside from the fact that I just wanted to get the data to a web browser.

 

libmongoose, is great, in that it is already written, and it seems to work really well. However, it does not seem to play well in the process memory "space". Functions I had written mysteriously stopped working properly, for no good reason. Just silently failing. After doing a bit of reading on libmongoose, this is, or can be expected behavior . . . great. I really did not want to go off and write my own http + websocket library. Passed that, Nodejs was simply out of the question. I wanted to keep my project C only.

 

So, I started reading on IPC in general. Everything, and anything I could get my hands on. It took me several days, a bit at a time to get a grasp on what all was available, and was leaning heavily towards Linux SYSTEM V Message Queues( for simplicity ), when I learned about the POSIX IPC counterparts.

 

Simply put. The POSIX IPC counterparts to SYSTEM V IPC simplify, and / or extend, or just make IPC in general so simple. Specifically what I would like to talk about is "shared memory". Ok, I must admit, I did know that is was relatively easy to setup a file in memory on Linux. I also knew about mmap, but when reading about SYSTEM V shared memory, and semaphores . . . my eyes were certainly glazed over. Way too complicated for a simple operation I was thinking. Then the fact that in order to create shared memory, and use mmap to access this memory required commits to disk( or so I thought, and this was out of the question ). I started imaging how to create a file in memory, and using mmap to access that. That is when I found out about POSIX shared memory. As it turns out, this is exactly what POSIX shared memory does . . .

 

So, before this "short story" turn into a novel, let me just share the very simple code I've come up with so far. Do keep in mind this is just the IPC "server" write stuff so far.

 

shm_test.c

#include "shm_test.h"
/*
open_shm_file(const char *fname), is just a helper function for shm_open(),
and ftruncate(). Perhaps making this source file less readable, but meant
to keep main() cleaner, and hopefully reducing running stack size. If file
does not exist, shm_open(), and ftruncate() are used to open, and set the
files initial size. Otherwise, the file is just opened. It is assumed that
file size is already set.

NOTE: open_shm_file() must be called in order to initialize shm_size.

Function source resides in shm_test.h
*/

int main(){

    int shm_fd = open_shm_file("acme");

    char *mmap_addr = mmap(NULL, shm_size, PROT_READ|PROT_WRITE, MAP_SHARED, shm_fd, 0);
    if (mmap_addr == MAP_FAILED)
        handle_error(strerror(errno));

    close(shm_fd);

    sprintf(mmap_addr, "%s\n", "Hello World");

    return 0;
}

shm_test.h

#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/file.h>
#include <sys/mman.h>

#ifndef SHM_TEST_H
#define SHM_TEST_H

#define FILE_PERMS 0644
#define handle_error(msg) do { perror(msg); exit(EXIT_FAILURE); } while (0)

long shm_size;

int open_shm_file(const char *fname){
    int shm_fd;
    shm_size = sysconf(_SC_PAGESIZE);

    const char *path = "/dev/shm/";
    char fpath[100];
    // Train wreck waiting to happpen. If you really want to break this
    // you will probably find a way.
    int len = sprintf(fpath, "%s%s", path, fname);
    if ( len < 0)
        handle_error(strerror(errno));
    
    int access_file = access(fpath, F_OK | R_OK | W_OK);
    if(access_file < 0){
        // File does not exist. So let's create it, and set initial size.
        if (errno == 2){
            shm_fd = shm_open(fname, O_CREAT | O_RDWR, FILE_PERMS);
            ftruncate(shm_fd, shm_size);
        }
        else { handle_error(strerror(errno)); }
    }
    else { shm_fd = shm_open(fname, O_CREAT | O_RDWR, FILE_PERMS); }

    return shm_fd;
}

#endif

One important thing to note when compiling. the -lrt compiler option *must* be used in order for the code to work / compile. -lrt == Linux Real time. Also note that for this code I did not use mlock(). But since the Beaglebone linux-image's come with swap disabled. We do not need to worry about the shared memory being paged to swap.

 

Compiling, and output:

william@debian-can:~/dev/test$ gcc -lrt shm_test.c -o test
william@debian-can:~/dev/test$ ./test
william@debian-can:~/dev/test$ cat /dev/shm/acme
Hello World
william@debian-can:~/dev/test$ du -h /dev/shm/acme
4.0K    /dev/shm/acme
william@debian-can:~/dev/test$ rm /dev/shm/acme

EDIT: Refactored code.

Link to post
Share on other sites

Ok, so if this makes things more interesting for others.

 

The problem I had with libmongoose, was that because of multiple function calls happening at unknown / unpredictable times.I was losing data in various variables due to function calls popping on / off the stack. Quite honestly, I'm not 100% sure how I could fix that( without spending a lot more time researching ), but more importantly I did not *want* to refactor my own code. Let alone spend days reading through 5000+ lines of code written by someone else. In short, I just  wanted it to work with the least amount of effort.

 

So here is the idea. You separate the project into two processes( 3 counting web client ). Each with it's own stack, and both using the same shared memory for communications. To be sure there are other potential issues that can arise. Foremost on my mind was a potential for shared memory race conditions. However I believe this can be dealt without using semaphores by creating one IPC server, and one IPC client.

 

From 30,00ft

 

The IPC server, reads in data ( from the CANBus in this case ). Parses the CANBus frames into NEMA 2000 fast packets. Based on the PGN values. Once a fast packet is constructed, it is then memcpy'd into a union / struct object in order to get at the actual data fields. Where then it becomes trivial to prepare the data for tranmission, and in this case, to a file in virtual memory.

 

The IPC client, which is also an http / websocket server runs in the background only using CPU when inbound connections are made. While also using a timing mechanism ( such as alarm() ) to come alive, read from this file in virtual memory, and transmit the data every time interval to each connected websocket client. This IPC client / http + websocket server *should* use very little CPU. Thus adding very little overhead to the whole project.

 

The http + websocket client( web browser, or not ) could use just about any web technology that is able to recognize this inbound data. IN this case, I've personally chosen to form JSON objects on the IPC server end, and JSON parse on the html / javascript ( web browser ) end. The IPC client / web server is just the "man in the middle" relaying data from IPC server, to web client.

 

Anyway, perhaps not the best use for IPC, but it should work fine for this situation. Without adding much if any noticeable overhead. However, another what I would think, a great use for this kind of IPC would be for permission boundary crossing. So say if you needed access to a file, or object that is normally only accessible from a privileged user( such as root ). You *could* use this technique to expose various object, and only the objects specified in code. Hopefully without compromising security, but that all depends on the implementation.

Link to post
Share on other sites

Sample output of where I currently am.

william@debian-can:~/dev/test$ gcc -lrt main.c -o test
william@debian-can:~/dev/test$ ./test
william@debian-can:~/dev/test$ cat /dev/shm/acme
{ "L3_rms_voltage": 239.31 }
william@debian-can:~/dev/test$ ./test
william@debian-can:~/dev/test$ cat /dev/shm/acme
{ "L3_rms_voltage": 239.02 }
william@debian-can:~/dev/test$ ./test
william@debian-can:~/dev/test$ cat /dev/shm/acme
{ "L3_rms_voltage": 234.64 }

 

Link to post
Share on other sites

So, I'm kind of having a mental block now. The downfall of many a project: Thinking it through as you go . . .

 

Current output:

william@debian-can:~/dev/test$ gcc -lrt main.c -o test
william@debian-can:~/dev/test$ ./test
william@debian-can:~/dev/test$ cat /dev/shm/acme
{ "L3_rms_voltage": 239.31, "L3_current": 16.30, "L3_frequency": 59.99 }
william@debian-can:~/dev/test$ ./test
william@debian-can:~/dev/test$ cat /dev/shm/acme
{ "L3_rms_voltage": 234.69, "L3_current": 00.10, "L3_frequency": 59.99 }
william@debian-can:~/dev/test$ ./test
william@debian-can:~/dev/test$ cat /dev/shm/acme
{ "L3_rms_voltage": 237.37, "L3_current": 15.90, "L3_frequency": 59.98 }
william@debian-can:~/dev/test$ du -h test
12K     test

So now, I'm left considering. "Well . . . great I have an object constructed for AC_OUTPUT_STATUS. Which is RMS volts, current and frequency. As far as interesting data goes anyhow. But what happens when I start transmitting multiple object types ?"

 

Maybe I need a JSON field like { "data_type": some_value, . . .}, and then switch off of some_value in javascript ? Does this sound reasonable ? Web development is not really my strong suit, and I normally just read about it as I go. But in this case I could possibly read a whole book or 3 on the subject, and still not be any closer to my end goal. Plus I really like javascript in some cases, but feel rather miffed that sometimes, it can feel like various values, and objects just seem to be pulled in out of thin air . . .

 

Anyway, any thoughts *anyone* has are welcome.

Link to post
Share on other sites

Well, it sort of works. The IPC server is definitely writing to the file faster than the IPC client can read it. I'll have to implement a "cheap" mechanism to make sure that data is read on the client side faster. Such as marking the first byte in the file as NULL or 0. That is, when the IPC client is done reading the values. That way I wont have to use alarm(), or hmm, maybe ualarm() will work . . .doubtful.

 

Fun fun fun.

 

https://picasaweb.google.com/lh/photo/z9liApnCXNiBar4YPbccRIyMIvEMRDv1YfiNp6Kf4kE?feat=directlink

 

EDIT: whooops, just managed to flood out, and crash firefox hah ! Who said websocket weren't fast ?

Link to post
Share on other sites

Ok, research time. Semaphores are honestly very overly complicated, and I did manage a very simple "file lock" mechanism between the two processes. However, it was very clunky, and I had problems with reading out the values( for this file lock ). One implementation in one process would not work in the other. For reasons unknown to me. Perhaps I was just too tired when writing the code, but . . .

 

So, what I propose is creating a data segment, and a simple binary file lock mechanism in one structure( struct ). So that this structure is just memcpy'd or perhaps even sprintf()'d into the mmap buffer when done. Of course toggling locked, before data manipolation, and unlocked when done. Then just a simple:

while(some_struct->file_lock == locked)
    usleep(2000);

Before attempting to access the mmap'd address region. Any thoughts ? To be continued . . .

Link to post
Share on other sites

I'd like some comments on this if anyone cares to comment.

#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/file.h>
#include <sys/mman.h>

#define FILE_PERMS 0644
#define LOCKED 1
#define UNLOCKED 0
#define handle_error(msg) do { perror(msg); exit(EXIT_FAILURE); } while (0)

struct shm_data {
    char *data;
    int file_lock : 8;
};

typedef union{
    struct shm_data sdata;
    char *mmap_addr;
}shm_t;

struct shm_data *new_shmdata(size_t size){

    struct shm_data *retVal = malloc(sizeof(struct shm_data));
    if(retVal == NULL)
        return NULL;

    retVal->data = malloc(size * sizeof(char));
    if(retVal->data == NULL){
        free(retVal);
        return NULL;
    }

    retVal->file_lock = 0;

    return retVal;
}

void del_shmdata(struct shm_data *shmdata){
    if(shmdata != NULL){
        free(shmdata->data);
        free(shmdata);
    }
}

int main(){

    const long shm_size = sysconf(_SC_PAGESIZE);
    int retval;
    // Open file as /dev/shm/acme with owner read / write permissions. File descriptor in shm_fd.
    int shm_fd = shm_open("acme", O_CREAT | O_RDWR, FILE_PERMS);
    if(shm_fd == -1)
        handle_error(strerror(errno));
    // Size the file, and fill with (null).
    retval = ftruncate(shm_fd, shm_size);
    if(retval == -1)
        handle_error(strerror(errno));
    // create data structure for data, and file locking mechanism. Size must be total file size - 1.
    // NOTE: shm_open() with ftruncate() will size the file to the nearest -> largest pagesize.
    // Based on requested size, and return value of sysconf(_SC_PAGESIZE). In my case - 4096.
    struct shm_data *ss = new_shmdata(shm_size -1);
    if(ss == NULL)
        handle_error("Unable to create data structure shm_data *ss.");
    // Encapsulate the data structure within a union to ease various operations. After which
    // *ss is no longer needed, so release *ss back to system memory pool.
    shm_t *shmem = (shm_t *)ss;
    del_shmdata(ss);
    ss = NULL;
    // Get the mmap pointer to the file, and link with the union "bufffer".
    shmem->mmap_addr = mmap(NULL, shm_size, PROT_READ|PROT_WRITE, MAP_SHARED, shm_fd, 0);
    if (shmem->mmap_addr == MAP_FAILED)
        handle_error(strerror(errno));
    // Since we have a pointer to the file. We no longer need the file descriptor.
    close(shm_fd);

    // Now we can use the file pointer as we would use any other (char *) pointer. With
    // The ability to add a file lock, so the file can reliably be used between multiple
    // processes. Simple demonstration below.

    // Check if file is locked. If file is locked, block process for 1/10th of a second.
    // Normally this would go into a program flow control loop.
    while(shmem->sdata.file_lock == LOCKED)
        usleep(100000);
    
    // If we're here the file is no longer locked. So lock the file, write( or read ) to the file.
    shmem->sdata.file_lock = LOCKED;
    sprintf(shmem->mmap_addr, "%s", "Hello World");

    // Unlock the file once done.
    shmem->sdata.file_lock = UNLOCKED;

    // What's in our file ?
    shmem->sdata.file_lock = LOCKED;
    printf("%s\n",shmem->sdata.data);
    printf("%d\n",shmem->sdata.file_lock);
    shmem->sdata.file_lock = UNLOCKED;
    printf("%d\n",shmem->sdata.file_lock);

    return 0;
}

output:

william@debian-can:~/dev/shm_test$ gcc -lrt -Wall test.c -o test
william@debian-can:~/dev/shm_test$ ./test
Hello World
1
0

 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...