Dynamic Linking of Shared Libraries in Go


Dynamic linking has taken a lot of flack from the go team, and they present some pretty irrefutable points, and yet I still feel like this is a necessary solution in some cases. Perhaps dynamic linking as we’ve all come to know it isn’t the correct answer, but a newer similar solution would be.

My proposal would be to design a style of dynamic linking that is based explicitly on function’s signatures rather than just symbols, just as is in static linking. If a black box is implemented to guaranty a certain output based on a given input, then it should be fair to hold whomever implements a library responsible for incompatible version changes. I would also like to see programmers take more care in ensuring the same library could be equally easily linked statically as dynamically–but that’s a different topic.

Justifications

Security: Take go’s fastcgi package for example. If you’re running a standalone instance using the fastcgi library, and a vulnerability comes out in that library, you will be required to upgrade every single package that utilizes it. To programmers who crave a world rid of shared libraries, that might not be a big deal, but imagine this from an intermediate sys admin’s perspective. Depending on your operating system’s distribution, you will have to either download new binaries for every one of these softwares, or else you will have to maintain sources for every single package you run in production, and remember to recompile all of them whenever this library is updated. I agree that in a lot of cases Dynamic linking is overkill and doesn’t save enough in disk space or can potentially use more memory and longer exec times, but this is one case where I’d rather see the library maintained separately from the solution.

Modular Applications: How would you go about writing something like the Apache Web Server, or the Pidgin Internet Messenger? Both of these applications have a daunting supply of official and user-contributed plug-ins made both efficient and possible by dynamic linking. The best alternative I can think of would be to rely heavily on some IPC techniques, which seems less efficient… but what do I know.

The Runtime:  I think this should be optional at compilation.  For example, Debian does a wonderful job maintaining consistent dependencies on glibc without breaking things, so why should that privilege be revoked?  Why would we want tons of 1 to 3 MB binaries for trivial operating system tools (echo, cat)?  On the other hand, statically linking the run-time (which I don’t see an obvious means to do in c++), does serve a great advantage for binary portability.  This provides a fantastic edge for proprietary manufacturers.

Memory

In my brief and naive research, I got the impression that one of the downfalls of dynamic linking is that the entire library is loaded into memory, whereas otherwise only the required components are.  And in the same article, it notes that shared libraries loaded by many applications can be loaded into memory just once.  While this of course still implies that the entire library is loaded into memory, this could still save some memory over time.  However, I wonder if this is only made possible by the fact that glibc is always loaded dynamically, and therefore it is able to coordinate that particular sharing of memory–or perhaps I’ve misinterpreted something altogether.

On the other hand, how necessary is it that we save that megabyte of RAM (or even disk space) this day in age?  This topic is a growing internal conflict of mine, but I still believe that making software as resource-efficient as possible should be a much higher priority than it is.  Any serious implementation should consider the possibility of running on small, low-power, low-memory embedded devices.  While flash memory is so inexpensive, there’s still a pretty active scene for modifying consumer wireless routers which lacks that resource.

All in all, I already see Go as a fantastic opportunity for developers who want to be able to write powerful and flexible applications on a smaller and more efficient development and run-time stacks.

 

Related reading:

taking a Go at plugin architecture

Advertisements
Posted in Programming | Leave a comment

TFTP GET transfer failing for some clients


This never dawned on me, but I spent too much time today figuring out why certain machines were unable to get files from a certain TFTP server. It turns out the “Certain TFTP Server” was not at all the issue. The tftp client (at least the one I’m using) opens up UDP port 34362 and waits for the server to drop data to it.

This of course means that you need to allow that traffic through the firewall.

Posted in Uncategorized | Leave a comment

Preseeding debian with software RAID (The kernel was unable to re-read the partition table … Invalid argument)


This is a problem I’ve been occasionally going back to for a couple months now. The problem is that I’m using preseed.cfg to perform a fully unattended debian installation via netboot and I keep hitting my head on this one obnoxious warning message “The kernel was unable to re-read the partition table on /dev/md2 (Invalid argument)”… which turns out doesn’t even matter. If you just hit “Ok” the installation goes through and succeeds just fine.

Today I spent hours researching preseed answers and all the processes and scripts involved in rendering these dialogs back to the user looking for any desperate means to abolish this message, when I got this other desperate idea…. cat /dev/mem > /mnt/mem . Yeah that’s right. I am going to personally read my RAM.

A quick search using less brought me to this section of the file:

@GO
T high partman/exception_handler_note
IPTION The kernel was unable to re-read the partition table on /dev/md2 (Invalid argument).  This means Linux won't know anything about the modifications you made until you reboot.  You should reboot your computer before doing anything w
ith /dev/md2.

And of course what stands out is high partman/exception_handler_note

A quick google search for “partman/exception_handler_note preseed.cfg” turns out a handful of samples where the line d-i partman/exception_handler_note note shows up in other folks’ configs. Why it was so difficult to track down this particular question for preseed is beyond me, but I suppose I’m just glad I didn’t give up on my roundabout detective work.

So the partition region of my preseed file looks like this:

d-i partman-auto/method string regular
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-md/confirm boolean true

d-i partman-auto/disk string /dev/sdb
d-i partman-auto/method string raid

d-i partman-auto/expert_recipe string                   \
      multiraid ::                                      \
              500 1000000001 1000000000 raid                    \
                      $primary{ } method{ raid }                       \
              .                                         \
              512 513 512 raid                          \
                      $primary{ } method{ raid }        \
              .                                         \
              512 513 512 raid                          \
                      $primary{ } method{ raid }        \
              .

d-i partman-auto-raid/recipe string             \
    1 1 0 ext3 /                                \
          /dev/sdb1                             \
    .                                           \
    1 1 0 ext3 /mnt/utility                     \
          /dev/sdb2                             \
    .                                           \
    1 1 0 swap -                                \
          /dev/sdb3                             \
    .

d-i partman/exception_handler_note note
d-i partman-md/confirm boolean true
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true

Beware…
Now of course I need to toss up some sort of disclaimer. The question “partman/exception_handler_note” is not very specific of course, and implies that this could be used for other unforeseeable problems. I’m comfortable using this because I’m in a controlled environment where I’m using the same distribution, installer version and same hardware across the board, but of course if you do this, you’re doing it at your own risk. I just have to say I’m so glad this is over.

Posted in Admin, Computer | Leave a comment

Dealing with SYSLINUX


Syslinux is simple and works great, but I wasted a lot of time today with a setup that refused to boot. Apparently if you install syslinux on a bootable partition, that just isn’t enough for the PC to boot.

By any means necessary find the mbr.bin file that came with your version of syslinux and cat it to the disk device (not the partition of course). For me “any means” meant a simple apt-get source syslinux. install-mbr simply did not work despite what debian’s guide said.

Most of what I know was given to me by rolling a custom debian installer. Of course I read all the way through their guide except the very bottom where there was a user contributed note expressing what I just explained. But I wanted to drop that note here just for the sake of polluting the Internet.

Here’s an example of what I’m working with for this project.

Files:

development:~/usb-disk# find .
.
./mbr.bin
./mkinstalldisk.sh
./boot-contents
./boot-contents/boot.txt
./boot-contents/initrd.gz
./boot-contents/vmlinuz
./boot-contents/initrd-autodisk.gz
./boot-contents/syslinux.cfg

And the script “mkinstalldisk.sh” … note that this is very crude…

#!/bin/bash
if whiptail --yesno "This is a dangerous tool... do you want to quit?" 8 40
then
    exit 1
fi

if whiptail --yesno "Are you SURE you want to wipe all data on disk $1 ???" 8 40
then

    # Destroy whatever's going on with this disk
    dd if=/dev/zero of=$1 bs=512 count=1

    # Add a FAT32 partition for the size of the disk
    echo -e "n\np\n1\n\n\nt\nc\na\n1\nw" | fdisk $1

    # Format....
    mkdosfs $11

    # Install SYSLINUX on this bootable partition
    syslinux $11

    # Install a valid MBR on the disk itself
    cat mbr.bin  > /dev/sdb


    # Add the boot environment

    mntdir=/mnt/target-`date +%s`/
    mkdir $mntdir
    mount $11 $mntdir
    cp `dirname $0`/boot-contents/* $mntdir
    umount $mntdir
    rmdir $mntdir

fi

Posted in Uncategorized | 1 Comment

Modify & Rebuild an initrd Image


Let’s assume your image is located at  ./boot-contents/initrd.gz

# Get into an empty directory
mkdir initrd-files
cd initrd-files
# Unpack the image
zcat ../boot-contents/ | cpio -id

Now make whichever changes you need.

# Rebuild the image
find . | cpio --create --format='newc' | gzip  > ../boot-contents/initrd.gz

Posted in Uncategorized | Leave a comment

Dynamic query filters for django


I’ve been stuck on this and looked around and found some fascinating & clever solutions, but the other day I realized how simple this really need be.

The problem: You have an undetermined number of constraints you may want to filter your database query by, and the Q object appears to require the programmer to have a pretty rigid query.

At least that was the misconception I started with.

The solution involves noting that the | operator is really just returning another Q object, which implies that you should be able to store that Q object and “or” some more Q objects onto later, just as easily as you could do inline.

Take this example:

myUsers = userInfo.objects.filter(Q(user=selectedUser[0]) | Q(user=selectedUser[1]) | Q(user=selectedUser[2]))

This example assumes we have a list of selectedUsers that we need more information about. But in the real world, you probably won’t know how long that list of selectedUsers really is.

So instead we can do something like the following:

usersQ = Q()
for sU in selectedUsers:
    usersQ = usersQ | (userid=sU)

And then you can go ahead and add usersQ to a chained filter on your query, and you will get the results you want. No fancy stuff.

Posted in Computer, Programming, Software, Uncategorized | Tagged , , , , | Leave a comment

Quick overview of (Asterisk) AGI using Python


I’m learning about AGI today, and naturally I decided to attack it with python and no library.

If you’re confused about what AGI can do, don’t be. As it’s name implies it’s not a terribly different concept from CGI, except for the fact that two way communication is extremely pivotal in AGI. Like CGI, It’s all stdin and stdout.

A crude explanation of AGI is a something like this:

  • Step 1. Your dialplan invokes an AGI script
  • Step 2. Your AGI script reads stdin until it finds an empty line, to get all the AGI variables (you needn’t get this confused with os.getenv — they’re all coming in through stdn in the format “property: value\n”)
  • Step (X). Your AGI script sends an AGI command through stdout
  • Step (X+1). Your AGI script reads a line on stdin to obtain the response and result from its last command (this is also how you allow your script wait for its command to complete).
  • Step (X+2). Do something with that result?
  • Finally, Your script exits.

This is what I came up with, the script simply announces your caller ID (as a number), plays a beep, and then accepts up to 5 digits, and then plays those digits back to you (as a number, again).

Extension (somewhere in extensions*.conf…):

exten => 234234,1,Answer
exten => 234234,n,AGI(andy.agi)
exten => 234234,2,Hangup

/var/lib/asterisk/agi-bin/andy.agi (+x):

#!/usr/bin/python
import sys,os,datetime

def log(msg):
    open('/tmp/andy.agi.log','ab+',512).write(
        "%s: %s\n"%(datetime.datetime.now(),msg)
    )

def get():
    res = sys.stdin.readline()
    log(res)
    res = res.strip()
    response,delim,result=res.partition(' ')
    result=result.split('=')[1].strip()
    result,delim,data = result.partition(' ')
    return response,result,data

def send(data):
    log(data)
    sys.stdout.write("%s \n"%data)
    sys.stdout.flush()

def getDigits(count=1):
    i=0
    digits=''
    digit = None
    while (i<count):
        i += 1
        send("WAIT FOR DIGIT 5000")
        digit = get()[1]
        if digit == "0" or digit == "-1":
            break;
        if digit:
            digits += str(int(digit)-48)
    return digits

AGIENV={}

env = ""
while(env != "\n"):

    env = sys.stdin.readline()
    log("Looping: env = \"%s\""%env)
    envdata =  env.split(":")
    if len(envdata)==2:
        AGIENV[envdata[0].strip()]=envdata[1].strip()

log(AGIENV)
log ("os.environ: %s"%os.environ)

send("SAY NUMBER %s \"\""%AGIENV['agi_callerid'])
res = get()
send("STREAM FILE beep \"\"");
res = get()
send('SAY NUMBER %s "*#"\n'%getDigits(5))
res = get()

Of course this is completely non-realworld. Anything going to log() was just for me to get a handle on what was actually happening, but I left it in here in case somebody else was going through the same thing. “tail -f” before or during the call gives you the best overview of what’s happening.

Other things

Arguments:

You can pass arguments to your AGI script from the dialplan.  Arguments are separated by the | character.

Environmental variables:

There are certain environmental variables you can access through getenv(), but they tend to be more related to asterisk and not so much AGI specifically.

There’s a great reference of agi variables, env variables, and agi commands available at voip-info, see my sources below.

I should point out that I would also likely be using something like pyst for my real-world applications, but I think it’s important to know all the nitty-gritty before jumping into a valuable library.

Sources:

Details:
  • Asterisk 1.6.2.18
  • Python 2.5
  • Debian 5
Posted in Uncategorized | 2 Comments